Close the gaps in a support assistant's security — decision exercise
Exercise: close the gaps in a support assistant's security
Original fictional scenario. No cloud account, API calls or paid services are required. Difficulty: intermediate · Estimated duration: 20 minutes
Fernway Telecom runs a customer-support assistant tuned on past chat transcripts. Its security lead answered a SAIF-style self-assessment. All names and findings are invented.
| Self-assessment question | Fernway's honest answer |
|---|---|
| Can you detect and remediate malicious or accidental changes to training data? | No — transcripts are loaded unchecked |
| Do you have a complete inventory of models and datasets? | Partly — two older tuned models are untracked |
| Do access controls prevent unauthorized reading or copying of models? | No — the tuning pipeline's service account is a project owner |
| Are the app and model protected against large-scale malicious queries? | Unknown |
| Are prompts and responses screened? | No — a customer recently pasted text that made the bot reveal internal notes |
Your decision
- For each "No", "Partly" or "Unknown", name the lifecycle stage and the risk.
- Name the Google Cloud control that addresses each gap.
- Choose the first two fixes and justify the order.
- Explain why fixing only the prompt problem would not be enough.
Rubric (10 house points)
- 3 points: stages and risks named correctly for all five rows.
- 3 points: a matching control for each gap.
- 2 points: a defensible order for the first two fixes.
- 2 points: a layered-defense argument for not stopping at one control.
Reference solution
- Training data unchecked → data stage, data poisoning → validate data before training.
- Untracked models → deployment, unknown assets → AI Protection inventory in Security Command Center.
- Owner-level service account → training, over-permissioning / exfiltration → least-privilege IAM roles (and Security Command Center to find over-permissioned accounts).
- Query floods unknown → serving, denial of service → Cloud Armor.
- Bot revealed notes → serving, prompt injection and data leakage → Model Armor screening prompts and responses.
A sensible first two: Model Armor (an active, customer-visible leak) and least-privilege IAM on the pipeline (the widest standing exposure). Data validation follows before the next tuning run.
Rejected alternatives. Fixing only prompt screening leaves poisoned training data, an over-powered service account and an unknown denial-of-service exposure — Google's guidance is never to rely on a single defensive measure. Buying a larger model fixes none of the gaps.
Sources and scope
All names and findings above are house-authored. The general principles are grounded in:
- https://docs.cloud.google.com/architecture/framework/perspectives/ai-ml/security
- https://cloud.google.com/use-cases/secure-ai-framework
- https://docs.cloud.google.com/iam/docs/overview
- https://docs.cloud.google.com/security-command-center/docs/security-command-center-overview
- https://docs.cloud.google.com/model-armor/overview