Measuring Model Trade-offs: Cost per Task, Effort and Evals

1.Compare cost per solved task, not price per token

A price list ranks models by cost per million tokens, but that number is not what you end up paying for. Anthropic's cost guide tells you to compare models "on cost per completed task, not per token." The reason is that models use different amounts of work for the same task. A stronger model may need fewer turns and less thinking. A cheaper model may fail more often and need retries. Even comparisons across versions can mislead: the same text costs about 30% more tokens on Claude Opus 4.7 and later, so a per-token comparison makes newer models look more expensive by construction.

The cheaper-looking option doesn't always win, and neither does the stronger one. On the same subset, Opus 5.5 at its default (medium) matched Fable 5.1 at its default, 92.8% against 92.3%, for about a fifth of the cost per solved task ($0.22 against $1.19). At the other end, Haiku 4.5 answered GPQA Diamond questions at about a fifth of Opus 5.5's cost per question, but with 63% accuracy against 92%, and it fell much further behind on long coding tasks. The guide's verdict on Haiku is that it "fits high-volume work with checkable outputs, not long agentic loops." The overall lesson: "The ranking flips by workload, and no price list tells you which way."

Staying on an old model costs money too. The guide says that if you are a model or two behind, "the cheapest lever is the model string." On Terminal-Bench 3, Opus 4.7, 4.8 and 5 each spent $8 to $15 per task but solved 7%, 15% and 41% of tasks. That brought cost per solved task down from $183 to $63 to $28. The more a workload defeats the old model, the more an upgrade saves per result.

Sources1

2.Tune effort before you switch models

The effort parameter controls how much thinking, tool calling and self-verification the model does. It applies to every output token, including text, tool calls and thinking, so it works whether or not thinking is enabled. The cost guide puts it this way: "Cost scales with all that activity; accuracy scales only with the part your task needs." Running a model at higher effort than the task needs buys depth the task never uses.

The cost guide's decision table maps each situation to a lever. If costs are too high and quality is fine, sweep effort down on your current model. If quality isn't good enough, "If you lowered effort, restore it; otherwise try the next tier up at low effort." The measurements back this up. On research benchmarks, low effort gave up 1 to 3 points for a third to a half off the cost per task, and in one case lowering effort beat an architecture change. Lower effort is also often faster, which matters when latency is the constraint. Long-horizon coding is the exception, where effort really does buy accuracy: on SWE-bench Pro, Opus 5.5 at low scored about 8 points below high, for about a third of the cost.

Defaults differ between models, and this is a common trap. Most models default to high, but Opus 5.5 defaults to medium. A request that omits effort therefore runs one level lower than it did on Claude Opus 5. For that reason the effort doc says to run an effort sweep on your own evals rather than carrying settings over from an earlier model. Effort is set per request with output_config, as in this sample from the effort doc:

client = anthropic.Anthropic()

response = client.messages.create(
    model="claude-opus-5-5",
    max_tokens=4096,
    messages=[
        {
            "role": "user",
            "content": "Analyze the trade-offs between microservices and monolithic architectures",
        }
    ],
     ? ={"effort": "medium"},
)

for block in response.content:
    if block.type == "text":
        print(block.text)

Sources12

3.Confirm the choice with evals, tail pricing and multi-model patterns

Every recommendation above ends with the same instruction: measure on your own workload. The selection guide says to create benchmark tests specific to your use case, and calls a good evaluation set "the most important step in the process." Then test with your actual prompts and data, compare accuracy, response quality and handling of edge cases, and weigh performance against cost. Anthropic's blog adds a reason public benchmarks fall short. Strong models such as Opus and Fable can solve almost every question on many tests (saturation), so the real differences only show up on tasks drawn from your own production work.

Which tasks you measure matters too. The cost guide says to "Price the tail of your workload, not the median". In other words, compare models on the hardest tenth of tasks. On typical tasks every model looks similar and the cheapest looks best. But a failed task still bills its tokens, then the retry, then whatever the failure costs downstream. On one 20-problem WideSearch run, two problems carried 43% of the spend.

Multi-model strategies are aimed at that tail. They pair a lower-cost model with a frontier model so that most tokens are billed at the lower rate. There are two shapes: an executor that escalates hard decisions to an advisor, and an orchestrator that delegates bulk work to cheaper workers. The guide attaches conditions to each. Add an advisor when a lower-cost model stalls only on hard decisions, and use an orchestrator when the work exceeds one context window. Before adding a second model, though, check the single-model effort curve first. In Anthropic's measurements, a multi-model setup that looked cheaper than the default single model cost more than that same model at lower effort.

Sources314

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.The model with the lowest per-token price is always the cheapest to run in production.Why is that wrong?

   Models use different amounts of work per task, and failed tasks still bill their tokens. Anthropic measured a model with a five-times-higher token price costing 35% less per solved task.

   Covered in Compare cost per solved task, not price per token

2. 2.Every Claude model defaults to high effort, so leaving effort unset on Opus 5.5 gives the same behaviour as on Opus 5.Why is that wrong?

   Claude Opus 5.5 defaults to medium, so a request that omits effort runs one level lower than it did on Claude Opus 5.

   Covered in Tune effort before you switch models