Output Evaluation and Validation — study note
Output Evaluation and Validation — study note
Domain 2 of the Claude Certified Associate – Foundations exam, and the heaviest domain. Its six objectives ask you to evaluate outputs for accuracy and completeness, spot hallucinations, inconsistencies and bias, fact-check, decide when human review is required, adapt output for its audience, and choose the right output format. This note summarizes each topic and the Anthropic pages behind it.
Checking accuracy
| Check | Why | What to do |
|---|---|---|
| One source, not the source | Claude can write things that look correct but are very mistaken | Scrutinize high-stakes advice before relying on it |
| Knowledge cutoff | Models may not know about events after their cutoff | Check answers about recent events against a current source |
| Calculations | Complex or mission-critical figures can be wrong | Verify with specialized mathematical software or by hand |
| Completeness | A partial answer is an incomplete answer | Check every part of the request was answered |
| Hallucinations | Quotes and precise details can look authoritative but be ungrounded | Trace precise details to a source |
| Inconsistency | Different answers to the same prompt can signal hallucination | Compare runs; ask Claude to verify its statements |
| Bias and missing context | A synthesis can drop context the originals contain | Correct it, and report biased or harmful output where the product allows |
Fact-checking techniques
- Open the cited sources. Every web-search response includes citations, and the original may hold context the summary dropped.
- A quote for every claim. Ask Claude to cite quotes and sources for each claim, and to retract any claim it cannot support.
- Stay inside the documents. Tell Claude to use only the documents you supplied, and give it permission to say it does not know. For long documents, ask for word-for-word quotes first.
These techniques significantly reduce hallucinations, but they do not eliminate them. Always validate critical information, especially for high-stakes decisions.
When a qualified professional must review
| Situation (Usage Policy) | Requirement |
|---|---|
| Consumer-facing advice or decisions directly affecting individuals in a high-risk area | A qualified professional in that field reviews before release |
| That output is presented directly to consumers | Disclose AI involvement, at a minimum at the beginning of each session |
| General wellness advice (sleep, stress, nutrition, exercise) | Outside the healthcare category |
The high-risk areas are legal, healthcare, insurance, finance, employment and housing, academic testing, accreditation and admissions, and media or professional journalistic content.
Anthropic suggests (legal work, not a Usage Policy requirement): keep a lawyer visibly directing the work, verify output against primary sources, and document the AI use. The existing duties of competence, confidentiality, supervision and verification still apply.
Shaping output
| Need | Approach |
|---|---|
| Right tone and detail | Name the audience and why a constraint matters |
| Different style | Describe the target style, not only what to avoid |
| Change one passage | In a doc or Markdown artifact, highlight it and use Edit with Claude |
| Compare directions | Edit an earlier message to branch the chat |
| Short answer | Inline |
| Substantial, self-contained, reusable content | An artifact |
| Output that must leave the chat | A downloadable spreadsheet, presentation, document or PDF |
| Comparison | Ask explicitly for a table with named columns |
| Data for another system | Structured data with the same named fields, curated to what the system needs |
Artifacts and file creation need the code execution and file creation capability turned on; organization owners can disable it.
Sources
- Claude is providing incorrect or misleading responses — https://support.claude.com/en/articles/8525154-claude-is-providing-incorrect-or-misleading-responses-what-s-going-on
- How up-to-date is Claude’s training data? — https://support.claude.com/en/articles/8114494-how-up-to-date-is-claude-s-training-data
- How does Claude handle mathematical equations and calculations? — https://support.claude.com/en/articles/10366421-how-does-claude-handle-mathematical-equations-and-calculations
- Reduce hallucinations — https://platform.claude.com/docs/en/test-and-evaluate/strengthen-guardrails/reduce-hallucinations
- Enable and use web search — https://support.claude.com/en/articles/10684626-enable-and-use-web-search
- Usage Policy — https://www.anthropic.com/legal/aup
- Using Claude for legal work — https://support.claude.com/en/articles/15707726-using-claude-for-legal-work-privilege-confidentiality-and-how-to-think-about-configuration
- Prompting best practices — https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/claude-prompting-best-practices
- Usage limit best practices — https://support.claude.com/en/articles/9797557-usage-limit-best-practices
- Claude’s response to my prompt is too brief — https://support.claude.com/en/articles/8114518-claude-s-response-to-my-prompt-is-too-brief
- What are artifacts and how do I use them? — https://support.claude.com/en/articles/17153992-what-are-artifacts-and-how-do-i-use-them
- Create and edit files with Claude — https://support.claude.com/en/articles/12111783-create-and-edit-files-with-claude