Close to Claude Sonnet 5 on the same corpus, at a fraction of the cost: what we measured

Luca Paper uses an existing language model. We add a corpus of official public texts, versioned and split by article and paragraph, and a multi-step drafting pipeline: corpus search, paragraph reading, plan, calculation, drafting, automated checks. A test bench, kept apart from drafting, judges every change. On that bench, our pipeline, on an inexpensive model, DeepSeek v4.1-flash, scores 7.62/10 on average over four runs (from 7.17 to 7.83). Claude Sonnet 5, working on the same corpus with our method, scores 7.83/10 on one run: on par with our best runs, slightly above our average (n = 6 per run). The pipeline’s model API cost is 8.3 to 16.2 times lower ($0.66 to $1.29 for the six memos depending on the run, at the prices billed on those days, against at least $10.71). Both configurations rest on the same corpus, as of 1 October 2026, where every citation can be checked at the source. The full comparison, with the make-up of the corpus, is published in Can AI draft a tax memo? What we measured.

Two caveats matter here: Claude’s cost is a lower bound, and the ratio depends on the pipeline’s cache rate, measured over the day.

We change the tool often, and every change is measured: a prompt, a drafting step or a new source can improve one memo and damage another, which a few examples are not enough to show.

Our ground rules

• —
  Frozen cases. A separate AI agent writes the cases: each case is a request for a memo on a French accounting and tax topic. Once written, the case set is never edited to let a change pass.
• —
  Prompt writers never see the cases. The instructions given to the model (the prompts) are written without seeing the cases, the memos produced during test runs, or their grades. An automated test fails as soon as a prompt shares a run of four words with a case; the prompt is then fixed, never the case. Prompts state principles, with no formulas lifted from a case.
• —
  An automated grader kept apart from drafting. An AI model, Claude Opus, which never sees our prompts and applies the same rubric to every memo. It gives a score out of 10, a verdict (sign as is, light edits, substantial rewrite, unusable) and a list of defects by class and severity. It can be wrong, and it has been (see our comparison with Claude).
• —
  Decision rules written, in principle, before the measurement. We first write down what will make us keep or revert the change, then we measure. Exception: the current rule (11 October 2026) was set after measuring the change it was then applied to. It judges a change against the range measured over several runs of the current version. That range comes from the version in place on 10 October 2026: run three times on the same six working cases, one run per case, its mean score ranged from 7.17 to 7.83 (n = 6 per run). A single run is therefore not enough to judge a change.
• —
  Memos are never hand-edited. What gets graded is exactly what the tool produced.
• —
  Working cases and control cases. The set is split into two halves. Six working cases serve to measure every change to our drafting, and six control cases serve to check that a gain does not hold only for the working cases (not yet done for the current version). The six working cases are those of the comparison between our pipeline and Claude.

Where we started: the same case does not get the same score twice

Our first measurement dates from 25 September 2026. It covers all 12 cases, three runs each, so 36 memos, with a mean score of 6.19/10 (n = 36). This does not compare directly with the scores on the six working cases. The grader is strict: at this starting point, before the changes measured since, it judged only one of these 36 memos signable as is.

The variability was already visible. The score of the same case varied on average by 1.75 points between the best and the worst of its three runs (n = 12 cases).

A change can hit its target and still make memos worse

We made the tool rely on a sheet of facts taken from the author’s request, for drafting as well as for verification. It hit its target: on the control cases, memos that questioned the facts the author supplied all but disappeared. But it degraded the rest. We measured it on all 12 cases, three runs each, and compared it with the starting point.

• —
  Memos became less consistent. On the working cases, the mean gap between the best and the worst run of the same case went from 1.83 to 3.0 points. The 1.83 is the starting point restricted to these six cases (1.75 over all 12).
• —
  On the control cases, memos with a major error on the applicable rule doubled, from 4 to 8 memos (6 control cases × 3 runs).

The change was reverted: the gain on one defect did not make up for the overall deterioration.

Pipeline economics

Here are the model API fees for each configuration, on the six working cases.

¹ At the 86.39% cache-hit rate (see “Tokens and costs” and the limitations). Without caching: $5.41.

² At the 85.31% cache-hit rate measured over the day of 11 October 2026, at the prices billed that day (see “Tokens and costs”).

³ See the limitations (“Claude + corpus tokens”).

Every row is graded by the same version of the grader. Our two runs are the ones whose cost was measured; over its four runs, our pipeline averages 7.62 (from 7.17 to 7.83, see our comparison with Claude).

The tokens, the prices used, OpenRouter’s activity record for each day and a script that recomputes the figures in this section are in the first article’s data pack. Two exceptions: the script reads back, without being able to redo them, the costs of the single-call Claude configurations, as reported by Anthropic, and the Claude + corpus token counts, taken from unpublished session logs.

Tokens and costs

A token is the unit of text, roughly a word fragment, that model providers bill for. For the six memos, the production pipeline called the model 322 times. It sent the model 20,303,957 tokens, and the model generated 564,137, or 3,478,016 tokens per memo, almost all of them sent rather than generated.

That volume comes from the pipeline working in a loop: it searches, reads a paragraph, computes, drafts. Every round, it re-sends the whole accumulated context to the model (the request, the texts already read, the plan, the draft). That context grows each round and goes out in full on every call. Most of the volume is therefore re-reading.

Caching cuts this cost sharply: the provider bills much less for text it has already received shortly before (a “cache read”). DeepSeek v4.1-flash prices at the Novita provider, per million tokens, taken from OpenRouter on 2 October 2026: $0.24 input, $0.0048 cache read, $0.96 output, which are the prices OpenRouter actually billed us.

• —
  At an 86.39% cache-hit rate, measured by OpenRouter over the day of 1 October 2026 (see the limitations), the six memos cost $1.29, or $0.21 per memo.
• —
  With no caching at all, the same work would cost $5.41.

Applied to the whole day, our prices and formula match OpenRouter’s bill to within 0.6%; the detail is in the downloadable data.

The run of 11 October 2026 called the model 204 times. It sent the model 12,549,131 tokens, and the model generated 326,886. At the 85.31% cache-hit rate measured that day and at the prices billed that day, the six memos cost $0.66; applied to the whole day, those prices match OpenRouter’s bill ($1.64).

Claude + Luca Paper corpus used 38,813,591 tokens for the six memos, about 6.5 million per memo. The Claude Sonnet 5 prices, taken from OpenRouter on 2 October 2026, are $2 input, $0.20 cache read and $10 output, per million tokens. At those prices, that comes to at least $10.71, 8.3 times the cost of our run of 1 October and 16.2 times that of the run of 11 October (see the limitations).

The gap comes partly from volume: Claude processed almost twice the tokens of our run of 1 October, and about three times those of the run of 11 October. It comes mostly from unit prices. In the cost computed for Claude, the bulk is cache reads, billed at $0.20 per million tokens against $0.0048 for DeepSeek. In the pipeline, those reads are only a small share. Its spend goes mainly to uncached input and to output, billed at $0.24 and $0.96 per million tokens, against $2 and $10 for Claude.

The single-call Claude configurations cost less than our pipeline: $0.69 for the six memos with web search, $0.37 with no tools (costs reported by Anthropic). But they get the lowest scores. A single call does not go and fetch the texts from the corpus, whereas the pipeline’s spend goes into that search. The quality gap points the same way, but we cannot attribute it entirely to that difference: our first grader, with no tools, had under-scored a configuration without the corpus, and the scores above come from its version that checks citations in the corpus (see our comparison with Claude).

These amounts are the model API costs of this bench run, that is, what the model provider bills. They are not what a Luca Paper user pays.

Method and limitations

• —
  Small samples. Six cases per half. The runs of the same version (7.17 to 7.83: three for our decision rule, four in the comparison with Claude) and the comparison of configurations cover the six working cases only, one run per case. The starting point and the fact-sheet change cover all 12 cases, three runs each. One or two cases are enough to move a mean.
• —
  Working cases. These are the cases on which we measure our own changes. The pipeline and its instructions were therefore tuned with their scores in view. The Claude + corpus configuration followed those same instructions and shares that advantage; the Claude configurations without the corpus do not.
• —
  A single grader, Claude Opus, which can be wrong. Every memo in the comparison was graded by the same version, which checks in the corpus the citations of memos without attached paragraphs.
• —
  Control cases. The six control cases exist, but neither the Claude comparison nor the current version of the pipeline has been measured on them yet.
• —
  Recognisable configurations. The grader can tell configurations apart: memos drafted with the corpus reach it with the cited paragraphs attached.
• —
  No human grading. No accountant graded these memos: the cases, the grading and the recheck are all automated.
• —
  Fact sheet. This change was measured together with an earlier adjustment, and we cannot separate their effects.
• —
  Connector and instructions. The Claude + corpus configuration used our MCP connector’s public tools, with our pipeline’s instructions and steps. Those tools do not include the chart-of-accounts lookup the pipeline has. We did not measure the connector alone, without those instructions.
• —
  Choice of accounts. We are working on the choice of PCG accounts in the proposed journal entries; in the meantime, that choice remains for the author to check.
• —
  Day-level cache rate. The 86.39% covers all our DeepSeek traffic on 1 October 2026, of which the bench made 322 of 542 requests, not the bench alone; the uncached cost is given alongside. Likewise, the 85.31% of 11 October covers the whole day, of which the bench made 204 of 479 requests. The ratios of 8.3 and 16.2 times depend on these rates and can move either way.
• —
  Prices. Those of the 1 October run were taken from OpenRouter on 2 October 2026; those of the 11 October run are the ones OpenRouter billed that day, matched by the day’s bill. They may have changed since.
• —
  Claude + corpus tokens. They were computed from session logs, each response counted once. The logs are not published. The output tokens recorded in them appear incomplete. The cost of this configuration is therefore a lower bound.
• —
  Figures recomputed. The figures were recomputed from the raw files (downloadable data).
• —
  Downloadable data: scores per anonymised case and defects per class, with a script that recomputes the figures on our decision rules. The tokens and costs are recomputed with the first article’s pack, linked above. They hold no case statements, no prompts and no grader comments. Case statements and our prompts are available on request at hello@lucapaper.com.