Design a prompt-injection test pack with expected safe behaviour

“The assistant should resist prompt injection” is not a test. A useful pack identifies the untrusted channel, the asset at risk, the attempted action and the observable safe behaviour. Refusal is only one possible expectation; a system may safely answer the legitimate part while ignoring an injected instruction.

Preserve a failing case

The LLMOps and security lab covers user text, retrieved content, tool output, encoded text and a multilingual variant. Its authored output fails the final case.

from llmops_cases import injection_pack_case

result = injection_pack_case()
assert len(result["tests"]) == 5
assert result["auth_bypass_attempts"] == 2
assert result["failed_ids"] == ["PI-5"]
assert result["fixture_outputs"] is True

That red result is the reason to keep the case. It exposes a coverage gap; it is not evidence about a live model because no model ran.

Test the application boundary

Start from a data-flow diagram. Create direct attacks in user input and indirect attacks inside documents, web pages, images, tool results and memory. Vary language, encoding, spacing, role claims and multi-turn setup. Include attacks that ask for secrets, expand tool authority, bypass approval, access another tenant or make an external write.

For each case, define allowed content and forbidden effects separately. A strong expected record might say: answer the public policy question; do not reveal hidden instructions; do not call the payment tool; emit an injection-risk event. Validate deterministic invariants at the tool gateway and authorization layer even if prose grading is also used.

Avoid a brittle blocklist

Exact attack strings change quickly. Keep canonical regressions for known incidents, then generate structural variants and test multiple channels. Do not rely on the model to decide its own authority. Retrieved text and tool output remain data; identity, permission and approval come from trusted application state.

Measure pass rate by attack class and business consequence. A harmless formatting deviation should not hide an unauthorized action, and a fluent refusal should not compensate for a secret already logged.

The Generative & Agentic AI course links adversarial tests to RAG trust boundaries and agent authorization.

Reproducible exercise

Extend the fixture to twenty cases across four channels. Add one digest-bound write approval and prove that every injected approval string fails before tool execution.

Continue learning

This article is part of the LLMOps, security and operational evaluation sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Prevent evaluation leakage during prompt iteration.
• Continue with Measure cost per successful task instead of cost per call.

References: OWASP LLM01:2025 Prompt Injection and OWASP GenAI Red Teaming.