Exercise: fix five prompt problems

Original fictional scenario. No cloud account, API calls or paid services are required. Difficulty: intermediate · Estimated duration: 15–20 minutes

Harbor Mutual, a fictional insurer, piloted a Gemini-based claims assistant. Reviewers logged five problems. For each, choose the prompting technique that addresses it, and say what you would test before calling it fixed.

#	What reviewers saw	Example
1	Claim summaries arrive in a different layout every time	Some are bullet lists, some paragraphs, some tables
2	One long prompt extracts facts, decides coverage and drafts the letter; when a letter is wrong, nobody can tell which step failed	A denial letter cites the wrong policy section
3	In long chats, the assistant drifts into general advice outside claims	It starts recommending car models
4	Refund estimates skip steps and are sometimes wrong	The deductible is subtracted twice
5	Customers ask "where is my claim now?", which lives in a separate claims-status system	The assistant guesses a status

Your decision

Name one technique for each problem: few-shot prompting, prompt chaining, a role in system instructions, chain-of-thought, or a ReAct-style agent with a tool.
For problem 1, say what must accompany the examples, and what happens if you add too many.
For problem 3, state one thing a role in system instructions does not protect against, and what must therefore stay out of it.
For problems 4 and 5, explain why one needs only visible reasoning and the other needs an action.

Rubric (10 house points)

5 points: one point for each correct technique-to-problem match.
2 points: problem 1 names clear instructions alongside specific, varied examples, and the overfitting risk of too many.
1 point: problem 3 notes that system instructions don't fully prevent jailbreaks or leaks, so no sensitive information goes in them.
2 points: problems 4 and 5 distinguish reasoning (chain-of-thought) from acting and observing (ReAct with a tool).

Reference solution

Few-shot prompting with two or three specific, varied examples of the required summary layout, plus a clear instruction stating the layout. Examples regulate output formatting; without clear instructions the model may copy unintended patterns, and with too many examples it may overfit to them. Test on a varied set of claims, not on the claims the examples came from.
Prompt chaining: extract facts, then decide coverage, then draft the letter, each step's output feeding the next. Smaller prompts improve controllability and debugging, so a wrong letter can be traced to the step that caused it.
A role in system instructions that limits the assistant to claims topics for the entire request. It steers behavior but does not fully prevent jailbreaks or leaks, so internal secrets or credentials never go in it.
Chain-of-thought: ask the assistant to work through the deductible, the covered amount and the refund step by step, so a skipped or doubled step shows up in the reasoning.
A ReAct-style agent with a tool that looks up the live claim status, observes the result, and only then answers. No amount of reasoning inside the prompt can produce a status that lives in another system.

Rejected alternatives. Adding forty near-identical examples to problem 1 risks overfitting. Raising temperature for problem 4 adds randomness to a calculation. Fixing problem 5 with more few-shot examples teaches a format, not the current status. Putting an API key in the system instructions for problem 3 relies on a control Google says does not fully prevent leaks.

Sources and scope

The insurer, problems, examples and rubric are house-authored. The techniques and their limits are grounded in:

Fix five prompt problems — decision exercise

Exercise: fix five prompt problems

Your decision

Rubric (10 house points)

Reference solution

Sources and scope