Evaluation, testing and optimization — study roadmap
Evaluation, testing and optimization — study roadmap
This unit is CCARP-U4 · Evaluation, Testing & Optimization, 16% of the exam blueprint. It carries six official objectives across three topics, and it is the domain where most candidates lose marks to plausible-sounding answers rather than to unfamiliar material.
| Topic | Objectives | What you must be able to do |
|---|---|---|
| T1 · Evaluation metrics and test frameworks | LO1, LO2 | Write criteria that can be failed, and build a set that can fail them |
| T2 · Testing and diagnosis | LO3, LO4 | Compare one variable at a time, and localise a fault before replacing anything |
| T3 · Optimization and monitoring | LO5, LO6 | Measure reuse before optimising, and instrument without over-collecting |
The five confusions this domain tests
Each of these appears more than once in the bank, because each is a place where a true statement is used to support a conclusion it does not carry.
- An average is not a contract. A candidate can raise the aggregate and fail the release, because a multidimensional contract is a conjunction of criteria and one of them is now unmet.
- A comparison with two moving parts attributes nothing. If the prompt and the model changed together, the result describes the bundle. That is still useful, and it is not evidence about either change.
- A missing fact is not a reasoning failure. Ask whether the supporting fact was in the evidence. Absent means restrict the model to its documents; present and mangled means look at effort and capability. The two look identical from outside.
- An input-token saving is not a cost saving. Prompt caching has no effect on output token generation, and a per-request figure excludes the requests that failed and were retried.
- A quiet collector is not a healthy system. Export errors fail silently by default and the agent runs on regardless.
Sequence
- T1 first. Everything downstream assumes criteria exist and that an evaluation set represents the workload. Both the A/B objective and the optimisation objective are unanswerable without them.
- T2 next. Its one-variable discipline is what makes any optimisation result in T3 believable.
- T3 last. It is the only topic where a change can look successful on the metric the team chose and be neutral or negative on the bill.
Budget roughly equal time across the three. T1 and T3 each carry more house-authored decision practice; T2 carries the diagnosis fork that the bank tests hardest.
What to carry into the exam
For each objective, be able to state the decision, the evidence that would settle it, and the claim that evidence does not support. The third is what the Professional-level items reward, and this domain is built almost entirely from true measurements paired with unjustified conclusions.
Sources and their limits
Every factual claim is grounded in retained Anthropic documentation, quoted verbatim with a recorded retrieval date and content hash. Two cautions:
- Prices and rates move. The cache write and read multipliers, the batch discount and the export intervals were current at the retrieval date recorded with each snapshot. The relationships they teach are stable; the numbers are not a quote.
- Beta surfaces change. The enhanced-telemetry variable and the span names it enables are documented as beta. Re-check before implementing.
Every threshold, latency figure, cost and score in the lectures, cards, questions and project is a house fixture, labelled where it appears. They exist to make a decision checkable and they are not vendor measurements.