Can AI draft a tax memo? What we measured

We had Claude draft the same six memos, with and without access to the texts, and ran them through our test bench. Claude is the only outside model we tested. The target deliverable is a first draft with the research done, the applicable rules, the proposed journal entries and, for every text cited, a link to the source; the author checks it, entries and choice of accounts included.

What makes a memo good is the corpus and the method, and both are ours. When Claude works on Luca Paper’s corpus following our method (our pipeline’s instructions and steps, one agent per case), it scores 7.83/10 on 6 cases, with no major defect, on one run. Our drafting pipeline applies the same method to an inexpensive model: 7.62/10 on average over four runs (from 7.17 to 7.83); its best runs match Claude’s score. Without the corpus, Claude stays around 6/10, with 6 or 7 major defects over six memos. These six cases are our working cases, on which the pipeline and its instructions were tuned (see the limitations).

Set-up

• —
  Six working cases. Each case is a request for a French accounting and tax memo, written and then frozen by a separate AI agent. The people who write the instructions given to Luca Paper’s model (the prompts) never read them.
• —
  An automated grader. An AI model, Claude Opus, which never sees our prompts and applies the same rubric to every memo. Each memo gets a score out of 10 and a list of defects: wrong rule, wrong citation, wrong accounting entry, invented fact, missing element, internal noise (the memo narrates how the tool works), digression. Each defect is rated major or minor. Every memo in the table was graded by the same version of the grader, which can check a citation in the corpus (see “Our grader got it wrong”).
• —
  One run per case and per Claude configuration, four for our pipeline (see the limitations).
• —
  Five configurations:

  1. Luca Paper’s drafting pipeline (production, versions of 1 to 11 October 2026, identical for drafting), on the DeepSeek v4.1-flash model: it searches the corpus, reads the paragraphs, plans, computes and drafts in several steps;
  2. Claude Sonnet 5 with Luca Paper’s corpus and our method. It reaches the corpus through our connector for Claude (MCP), which gives it corpus search, paragraph retrieval and the calculation engine. Each case is handed to a separate Claude agent, which receives our pipeline’s instructions and follows its steps one by one: scoping, research, plan, drafting;
  3. Claude Sonnet 5 with web search, in a single call;
  4. Claude Sonnet 5 with web search and three self-review passes: it reviews and corrects its own memo three times;
  5. Claude Sonnet 5 with no tools, in a single call.

The corpus, as of 1 October 2026, holds French accounting and tax texts, including, on the accounting side, the PCG (the French chart of accounts and accounting rules), the regulations of the ANC (the French accounting standard setter) and the Code de commerce (French Commercial Code) and, on the tax side, BOFiP (the tax authority’s official guidance), the CGI (French General Tax Code), the CIBS (Code of taxes on goods and services) and the LPF (Tax Procedure Code).

Results

A major research error is a wrong rule, a wrong citation, a missing element or an invented fact rated major: research is what you first expect from a colleague preparing a memo. With the corpus and our method, Claude has none over 6 memos and our pipeline 2 over 24. Without the corpus, Claude has from 0 to 3 out of 6 depending on the configuration, and its memos carry far more major defects.

What it shows

The 7.83 was obtained with the corpus and our method together: Claude was plugged into the corpus and also followed, case by case, our pipeline’s instructions and steps (see the limitations). Without the corpus, the same Claude stays around 6 (from 6.0 to 6.17), with 6 or 7 major defects over six memos. The corpus and the method add almost two points; on that run, it has no major defect.

Our pipeline applies that method with a much cheaper model, DeepSeek. Over its four runs it averages 7.62: its best runs match Claude’s score (7.83), the others stay slightly below. It sometimes has a weak run on one case (5/10 at the lowest) and has major defects in three runs out of four; Claude’s run had none. Two very different models therefore get close scores once they have the corpus and the method: the corpus and the method seem to matter more than the model.

That level takes a lot of reading: for each memo, Claude + corpus read almost twice as much text as our run of 1 October, and about three times as much as the run of 11 October. The volumes and their cost are set out in Close to Claude Sonnet 5 on the same corpus, at a fraction of the cost: what we measured.

A source behind every rule

The pipeline cites every text precisely: the article of the CGI or the PCG, the paragraph of BOFiP. Example: in Amortissement dérogatoire : pourquoi la comptabilité et le fiscal divergent (in French), every citation of article 39 of the CGI leads to Légifrance. These example pages are raw answers from Luca Paper (question-and-answer mode or drafting pipeline), never edited by hand; they are not among the six graded memos.

Our grader got it wrong

Our first grader ran with no tools. For the memos drafted with the corpus, the cited paragraphs were attached to the memo, so it could compare them with the text. For the configurations without the corpus, no text was attached: it judged citations from memory.

The problem surfaced on Claude with web search and three self-review passes. Reviewing the grader’s reports on its memos, we spotted correct citations counted as wrong, mostly when the memo itself flagged a source as unchecked. Claude agents that, unlike the grader, had access to the sources rechecked its six memos; no person reviewed them. Its score went from 5.83 to 6.83, and the recheck found a real omission that the grader had missed.

So we gave the grader a tool that checks a citation in the corpus when no text is attached, then graded every memo in the table again with that version. The scores without the corpus went up a little: from 5.5 to 6.0 with no tools, from 6.0 to 6.17 with web search, from 5.83 to 6.17 with three self-review passes. So did those with the corpus: from 7.5 to 7.83 for Claude, from 7.33 to 7.83 for our pipeline’s run of 1 October. The agents’ recheck is a different method: its 6.83 is not in the table.

Method and limitations

• —
  Six cases, a single automated grader (Claude Opus), with no human grading. One run per Claude configuration, four for our pipeline. That is too few for any precise statistical conclusion, and small gaps mean nothing: from one run to the next, the same version of our pipeline scores from 7.17 to 7.83 on these six cases (see our test method). With a single run, we do not know how much Claude varies.
• —
  Working cases. The six cases are the ones on which we measure our own changes: the pipeline and its instructions were tuned with their scores in view. Claude + corpus followed those instructions and shares that advantage; the Claude configurations without the corpus do not.
• —
  Grader. It may err both ways: counting a correct citation as wrong, or missing a real error. Its tool checks citations in our corpus, not on the web.
• —
  Corpus and method measured together. We did not separate the effect of the corpus from that of the method, and DeepSeek was never measured without the corpus. With two models and six cases, we draw no general rule from it.
• —
  Method, not just the connector. The Claude + corpus configuration used our connector’s public tools and, on top of them, our pipeline’s instructions and steps, one agent per case. Those tools do not include the chart-of-accounts lookup the pipeline has. We did not measure the connector alone, without our method, and there is no evidence it would reach the same score.
• —
  Claude version. Every Claude configuration uses Claude Sonnet 5, measured on 2 October 2026.
• —
  Detailed method, volumes and costs: the test protocol and the economics of each configuration are set out in our test method.
• —
  Downloadable data: scores per anonymised case, defects per class, the volumes processed and a recompute script. They hold no case statements, no prompts and no grader comments. Case statements and our prompts are available on request at hello@lucapaper.com.