A pricier model for the plan step: what it brings, what it costs

We handed the plan step of our drafting pipeline to a stronger and pricier model, GPT-6.1 Terra (OpenAI), in place of DeepSeek v4.1-flash. The plan is the call that decides, before drafting, which rules and which journal entries the memo will contain. On our six working cases, the two runs with this plan reach or exceed 8/10 (8.00 and 8.33), against 7.83 and 7.50 for two runs on DeepSeek alone, that is 8.17 against 7.67 on average: a gap of 0.5 points. The improvement shows in the scores, but it stays at the high edge of our benchmark’s noise, and it costs a lot more: $0.27 per memo against $0.10, that is 2.6 to 2.7 times more.

Three caveats matter here: the gap is at the high edge of the noise, the pipeline’s code also changed between the two series of runs, and our automated grader drifts from one session to the next. They are detailed below.

What we changed

Our drafting pipeline searches the corpus, reads the paragraphs, plans, calculates and drafts in several steps (see Close to Claude Sonnet 5 on the same corpus, at a fraction of the cost: what we measured). The plan rereads all the sources the search gathered, twice per memo: once, then a second time after asking for more texts to be read, that is 12 calls for six memos. With an inexpensive model this is the slowest step of the pipeline, because the model reasons at length. We tried a stronger model that answers faster but costs far more per token (a token is the unit of text the model’s provider bills). Everything else (search, reading, drafting) stayed on DeepSeek v4.1-flash. Between the runs on DeepSeek alone and these, the pipeline also received ten other improvements, described below: the measured gap therefore mixes the two.

What we measured

These are the six working cases of our other articles, the same grader. A run is a series of six memos, one per case.

Means: 8.17 for the 12 memos with the plan on Terra, 7.67 for the 12 memos on DeepSeek alone. None of the 12 memos with the plan on Terra needed a substantial rewrite, against 2 of the 12 memos on DeepSeek alone. Journal-entry defects fall from 9 to 2 over 12 memos, which fits the plan’s role, since it decides the entries. Invented facts rise from 3 to 5 and language or formatting defects from 1 to 4; the other defect types stay close (see the data pack). The plan is also much faster: 39 s on average per call against 172 s, and about 5,600 answer tokens, reasoning included, against 31,000.

An improvement at the high edge of the noise

• —
  What the noise allows. Two runs of the same model differ by 0.33 points, for Terra as for DeepSeek, and an unchanged version of our pipeline scored from 7.17 to 7.83 over three runs. Our two runs with the plan on Terra, at 8.00 and 8.33, sit at the high edge of that range, or slightly above it. Under the rule we apply to our changes (a gain must exceed the range), this would count as progress; we nonetheless present it with caution, for the reasons below.
• —
  Two runs against two. Both runs on Terra are above both runs on DeepSeek. If the runs were interchangeable, the chance of getting this ranking by luck would be about one in six. Defects also move from one run to the next: journal-entry defects are at 4 and 5 on DeepSeek alone, at 2 and 0 with the plan on Terra.
• —
  A grader that drifts. In every session the grader scores six reference memos whose expected score is known; its mean gap to that expected score (positive when it is more lenient) was +0.50 and +0.17 for the two runs on Terra, 0.00 and −0.33 for the two runs on DeepSeek: the same order as the scores. Once that gap is subtracted, the four runs fall back to about 7.83. With only six reference memos this is not a correction, but it shows that the grader’s drift can produce a gap of this size on its own.
• —
  Code that changed. Between the DeepSeek runs (27 October) and the Terra runs (28 and 29 October), ten changes were made to the pipeline, all meant to improve quality. We do not know how much they contribute to the gap. Our reading is that the plan model contributes, but we did not rerun DeepSeek with the current code to isolate it: the measured gain mixes the two.

What it costs

The prices below are those of the latest run on Terra, the one where the model’s cache was set up (see the next section). They are the amounts billed by OpenRouter, the platform through which we call the models, for the six memos of each configuration, without the grader.

The plan accounts for 69% of a memo’s cost. It reads close to 100,000 tokens per memo, at a per-token price 6 to 8 times DeepSeek’s ($1.20 per million input tokens and $6 for output, against $0.195 and $0.78). On the first run on Terra, before the cache was set up and with a retry call, a memo cost $0.32, that is 3.1 to 3.2 times the price of DeepSeek alone. With the plan on Terra, the pipeline stays at least 6.6 times cheaper than Claude with the corpus (at least $10.71 for the six memos, comparison detailed in the article cited above).

The cache: what the bill taught us

Prompt caching avoids paying full price for texts already sent. With this model, our plan never read from it, and nothing in our logs showed it.

• —
  An invisible 25% surcharge. The listed price is $1.20 per million input tokens. As long as nothing is read from the cache, every token is billed as a cache write (a token stored to be reused), $1.50, 25% more. We found it by comparing each call’s cost with its token count.
• —
  An extended text is not read from the cache. The second plan repeats the first and adds lines, but by default the provider only marks a reuse point at the end of the last message: there is nothing to reuse. Our tests show it (see the data pack): the same text sent twice is read almost entirely from the cache, an extended text is not read at all, and changing the reasoning setting between two calls also cancels the reuse.
• —
  Explicit reuse points. We now place these points ourselves, on the plan call only, so that the second plan reuses the first.
• —
  Result on the memos: 40%. The second plan read 127,498 of the 318,639 tokens of its text from the cache, that is 40%, against 91% in a clean test, which is 21% of everything the plan reads. At this level the reuse points cost the same as switching the cache off, to within $0.005 over the six memos: you have to exceed 21% to gain from them. We have not checked why the rate is lower; our hypothesis is that lines already sent get longer between the two plans.

Between our two runs on Terra, the plan’s cost fell from $1.51 to $1.12. The cache explains $0.18 of that drop, the retry call of the first run $0.11, and shorter starting texts the rest ($0.09). The cache is set up and it works, but at this rate it only brings the bill back to the listed price, without taking it below.

Method and limitations

• —
  Six working cases, one grader, few runs. Two runs per model, one per case, six memos each. These are the cases on which we tune our pipeline and its instructions: the scores should be read with that bias.
• —
  Different code. Ten changes separate the DeepSeek runs from the Terra runs. We did not rerun DeepSeek with the current code: the 0.5-point gap therefore mixes the effect of the plan model and that of those changes.
• —
  A single grader, Claude Opus, which can be wrong and whose gap to the expected score varies from one session to the next (see above). No person graded these memos.
• —
  Who read what. The pipeline’s instructions are written by a person who reads neither the graded memos nor their evaluations; for this article, only the grader’s summary tables were read.
• —
  A single model compared for the plan. Nothing allows us to generalise to other models or to other steps of the pipeline.
• —
  Prices. The amounts are those OpenRouter billed, request by request, for each run’s time window; our API’s log and the bill agree to within $0.0002. The cost of DeepSeek alone is that of the first run, given as a range because the last minute of its window also contains the first calls of the next run. Prices may have changed since. Claude with the corpus is a lower bound (see the article cited above).
• —
  Example page. One page on the site exists in both versions, untouched: Créance client irrécouvrable : traitement fiscal de la perte (in French; current version: plan on Terra; earlier version: DeepSeek alone). It is not one of the six memos evaluated.
• —
  Downloadable data: scores per configuration, defects per class, the plan calls with their tokens and cost, the prices, the bill per time window and the cache tests, with a script that recomputes the article’s figures. They hold no case statements, no prompts and no grader comments. Case statements and our prompts are available on request, at hello@lucapaper.com. Data pack.