Evaluation, testing and optimization — reference

A decision-table companion. Each table names a decision, the documented input that settles it, and the claim it does not license.

1. Success criteria (LO1)

Quantifying a hazy criterion needs three things: a quantity, the population it is measured over, and the instrument that decides. The documented example replaces "safe outputs" with less than 0.1% of outputs out of 10,000 trials flagged for toxicity by the content filter.

The documented common criteria are task fidelity, consistency, relevance and coherence, tone and style, privacy preservation, context utilization, latency and price. The list is explicitly non-exhaustive, and most use cases need multidimensional evaluation along several of them.

⚠ Metrics and methods are different things. Metrics: F1, BLEU, perplexity, accuracy, precision, recall, response time, uptime. Methods: A/B testing against a baseline, user feedback such as completion rates, and edge case analysis as the share handled without errors.

2. Evaluation design (LO2)

Edge case classes named: irrelevant or nonexistent input; overly long input; poor, harmful or irrelevant user input in chat; and ambiguous cases where even humans would struggle to agree. That last is to be included, not removed.

Two refinements for a model grader: a rubric detailed enough that the verdict is mechanical, and reason first, then discard the reasoning before scoring. Ask for empirical or specific output, since purely qualitative evaluations are hard to assess at scale. A use case may require several rubrics.

3. A/B testing and rollout (LO3)

A/B testing compares against a baseline model or an earlier version. If two things moved, the result describes the bundle.

Before the rollout, record: a threshold on each criterion, a rollback trigger, and the owner of the call. Criteria that arrive after the number are not criteria.

⚠ An aggregate gain does not discharge a separately named criterion. Re-weighting the aggregate afterwards is changing the contract, and that decision belongs to whoever owns it.

4. Diagnosis (LO4)

Localise before replacing. Ask for step-by-step reasoning to expose faulty logic; repeat the same prompt and compare, since inconsistency can indicate hallucination; require a supporting quote per claim and retract where none exists.

⚠ These techniques significantly reduce hallucinations but do not eliminate them. Critical information still needs validation, especially for high-stakes decisions. If a downstream action is irreversible, the design needs a check that does not depend on the model having been right.

5. Optimization (LO5)

Rates as documented on 2026-09-05. The relationships are stable; the figures are not a quote.

The cached prefix follows tools → system → messages. A change invalidates that level and every level after it, so editing a tool definition invalidates the entire cache. Cache hits require 100% identical segments up to the breakpoint. Place the breakpoint on the last block that stays identical across requests.

⚠ The silent failure: a prompt under the minimum cacheable length is processed without caching and no error is returned. Verify with the usage fields — if both cache_creation_input_tokens and cache_read_input_tokens are 0, it was not cached.

Batching suits work with no immediate-response requirement: most batches finish within an hour, cutting cost by 50% and raising throughput (documented 2026-09-05). Large-scale evaluation is a named fit. It does not improve per-request latency.

⚠ Cost per successful task, not per request. Input-token spend excludes output tokens, which caching does not affect, and a write costs 1.25× the input it replaces — a prefix never read again is worse off cached. (Cached prefixes also still occupy the context window, but that is capacity, not price.) And the SDK's cost total is a client-side estimate from a bundled price table — the documentation says not to bill end users or trigger financial decisions from it.

6. Monitoring (LO6)

Intervals as documented on 2026-09-05.

Telemetry is off until enabled with at least one exporter chosen.

⚠ Export errors fail silently by default. The agent runs on and the telemetry is dropped. A quiet collector is not evidence of health; verify arrival.

Structural by default: durations, model names and tool names on every span, and token counts when the API request returns usage data — so failed or aborted requests may omit them. Content is opt-in, escalating to raw bodies carrying whole conversations. Leave them unset unless the pipeline is approved for that data.

⚠ Attribution needs deliberate injection. Default identity attributes name the calling credential, not the end user. Inject end-user identity as resource attributes to make tool decisions and MCP activity a per-user audit trail a SIEM can consume. The session identifier, carried by default, groups calls into one conversation but says nothing about who.