Stakeholder communication and lifecycle — reference
Stakeholder communication and lifecycle — reference
A decision-table companion. Each table names a decision, the documented input that settles it, and the claim it does not license.
1. Requirement or decision (LO1)
| Sentence | What it really is | What it lets you check |
|---|---|---|
| "The assistant should be accurate" | A wish | Nothing |
| "Use the fastest available model" | A decision | Nothing |
| "Use this named model" | A decision | Nothing |
| "F1 of at least 0.85 on a held-out set" | A criterion | Whether the release passes |
Building a successful application starts with clearly defining success criteria and then designing evaluations to measure performance against them. The documented improvement is exactly this move: "good performance" becomes "accurate sentiment classification", and "the model should classify sentiments well" becomes an F1 score of at least 0.85 on a held-out test set.
The properties a criterion carries
| Property | The documented demand |
|---|---|
| Specific | Say what you want to achieve, not that you want it to be good |
| Measurable | Quantitative metrics, or well-defined qualitative scales applied consistently |
| Achievable | Based on benchmarks, prior experiments, research or expert knowledge |
| Relevant | Aligned to the purpose of the application and the needs of its users |
⚠ Achievable is the one that gets skipped. Success metrics should not be unrealistic to current frontier model capabilities. A threshold nobody can hit is not ambition; it is a criterion built to fail.
⚠ One criterion is a finding, not a result. Most use cases need multidimensional evaluation along several success criteria, and the published list — task fidelity, consistency, relevance and coherence, tone and style, privacy preservation, context utilization, latency, price — is explicitly non-exhaustive.
2. Decisions and trade-offs (LO2)
| Factor | The documented question |
|---|---|
| Capabilities | What specific features will you need the model to have |
| Speed | How quickly does it need to respond in your application |
| Cost | What is the budget, for development and production usage |
| Effort | Trades intelligence for latency and cost within a single model |
⚠ Tuning effort is often a better lever than switching models. A decision record that only weighed model swaps has left out the cheaper alternative.
Levers, and what each one trades
| Lever | Gained | Given up |
|---|---|---|
| Lower effort | Latency and cost | Intelligence within that model |
| Fast mode | Up to 2.5x higher output speed | Premium pricing |
| Multi-model pairing | Most tokens billed at the lower rate | An orchestration boundary to maintain |
A defensible record carries the decision, the alternatives, and the evidence. The documented evidence step is to create benchmark tests specific to your use case — having a good evaluation set is the most important step in the process — then test with your actual prompts and data, then weigh performance and cost trade-offs.
⚠ Anthropic token cost typically dominates container infrastructure cost by an order of magnitude or more. A team optimising the hosting bill is working on the smaller number.
3. Service levels and expectations (LO3)
| Measurement | What it covers |
|---|---|
| Baseline latency | Prompt processed and response generated, ignoring tokens per second |
| Time to first token | From prompt sent to the first token of the response |
⚠ Name the metric before agreeing the number. These are different intervals over the same request, and on a long answer they differ by a lot.
A service-level line needs three parts: the named metric, a threshold with the share of requests it covers (the documented multidimensional example attaches a percentage to its response-time criterion rather than stating a bare average), and a note of when it was measured. It is always better to engineer a prompt that works well first and then try latency reduction, because reducing latency prematurely might prevent you from discovering what top performance looks like.
Whose lifecycle governs
| Where you deploy | Whose dates apply |
|---|---|
| Claude API | Anthropic's first-party commitment |
| Claude Platform on AWS | The first-party model lifecycle |
| Microsoft Foundry | The Claude API lifecycle schedule |
| Amazon Bedrock | Bedrock sets its own lifecycle dates |
| Google Cloud | Google Cloud sets its own lifecycle dates |
⚠ Every model ID, whether dated or dateless, has its own distinct deprecation and retirement schedule. The published date is a floor ("not sooner than"), not an appointment.
What a pinned model ID freezes
| Fixed for the lifetime of the ID | Free to change |
|---|---|
| Weights | The request router |
| Configuration | Safety classifiers |
| (Updates ship under a new ID) | Sampling logic |
⚠ Infrastructure updates occasionally produce minor differences in observable behavior even when the model ID and weights have not changed, and where you see unexpected behavioural differences on a previously stable ID, an infrastructure update is the most likely cause. A dateless ID is a pinned snapshot, not an evergreen pointer — the documentation names the opposite belief as a common misconception.
4. What a handover document answers (LO4)
| Decision | Why it is pre-launch |
|---|---|
| Session and state persistence | Default local disk is lost on restart, scale-down or a move to another node |
| Observability | Without telemetry you cannot see which tools ran, how long they took, or where a session stalled |
| Auth and secrets | Inbound at a gateway; outbound credentials injected by a proxy after the request leaves |
| Scaling and concurrency | Bounded by how many subprocesses host RAM can hold |
| Cost | Token cost dominates infrastructure cost |
| Multi-tenant isolation | Shared filesystem state can leak one tenant's context into another's session |
Known limitations to plan around
| Limitation | The documented answer |
|---|---|
| No top-level session timeout | Bound the tool-use round trips with a maximum turn count |
| Memory growth over long sessions | Cap session length or recycle subprocesses periodically |
| Wide parallel-subagent fan-outs can hit rate limits | Break work into smaller batches |
| No per-subagent wall-clock deadline | The stall watchdog fires on no output; it is not a total-runtime deadline |
⚠ The mirror error is the failure that leaves the run looking fine. When the SDK cannot deliver a batch to the store it drops the batch and continues the query — success, exit zero, record gone. Alert on these if store durability matters. Note the conditional: it is a design decision, not a default.
⚠ A session store mirrors transcripts only, not memory files or other working-directory artifacts, and the subprocess writes to local disk first with the store receiving a forwarded copy.
⚠ The published per-session memory figure is a floor, not the ceiling. Measure yours with a representative session at your target length under expected tool load, recording peak resident memory.
5. Lifecycle phases (LO5)
| Boundary | What must cross it | What is lost if it does not |
|---|---|---|
| Design → handoff | Decisions, constraints, known limitations, environment assumptions | The team rediscovers each one during an incident |
| Handoff → monitoring | Owners, alerts, and which cost figure is authoritative | Nobody is paged, and an estimate is read as a bill |
| Monitoring → iteration | The failure, as an automatically gradable eval case | The same defect ships again |
⚠ It works on my machine is an ordinary failure, not an exotic one. The documented first suspect for Python is that a container or service manager runs the application with a different path than your shell, so a local install is invisible to the process.
Which cost number goes in front of whom
| Question | Where the answer comes from |
|---|---|
| How is this run trending | The agent result's dollar fields |
| What are we actually billed | The Usage and Cost API, or the Console usage page |
| What do we charge a customer | Not these fields at all |
⚠ The total_cost_usd and costUSD fields are client-side estimates, not authoritative billing data. They are computed locally from a bundled price table and can drift from what you are actually billed. Use them for development insight and approximate budgeting; the documentation says plainly: do not bill end users or trigger financial decisions from these fields.
Making the estimate honest anyway
| Read from | What it costs you |
|---|---|
Per-step output_tokens | A placeholder reported before the response was generated |
Raw usage on a budget error | Leaves out the response that crossed the budget |
| Successful results only | Failed conversations still consumed tokens up to the failure |
total_cost_usd or modelUsage | The documented basis, where the choice exists inside the result |
⚠ A crash emits a final error result that may carry every cost field zeroed; recover the totals from what arrived before it.
⚠ Tracing is in beta, and span names and attributes may change between releases. A dashboard is a promise about a schema.