A living collection of failure patterns discovered while evaluating AI agents in adversarial scenarios, along with engineering insights and product implications.
Rather than documenting isolated bugs, this repository focuses on identifying recurring failure patterns, understanding why they occur, and evaluating their impact on production AI systems.
Core belief: knowing how an agent fails is what lets you make it fail less. Every failure below is paired with the underlying pattern it reveals, because the pattern — not the specific bug — is what transfers to your next agent.
Finding bugs is only part of building reliable AI systems.
Equally important is understanding:
- Which failures are critical to fix
- Which failures can be mitigated through prompt design or guardrails
- Which require additional evaluation before making product decisions
- Which are acceptable trade-offs
This repository captures both the engineering perspective and the product management perspective for every observation.
A Retrieval-Augmented Generation (RAG) support agent built for a fictional company (Vertex Cloud Technologies) with support ticket escalation capabilities.
The knowledge base intentionally contains:
- Missing information
- Contradictory information
- Escalation rules
- Edge cases
This makes every response measurable against a known ground truth.
Rather than relying on scripted conversations, testing is performed using adversarial interactions designed to expose reasoning, retrieval, and tool-calling failures.
Observed behaviors are re-tested across multiple sessions before being considered recurring patterns.
The current taxonomy continues to evolve as new failure modes are discovered.
| Category | Description |
|---|---|
| Input Parsing | Incorrect interpretation of user input |
| Reasoning | Correct understanding but incorrect conclusions |
| Tool Usage | Incorrect tool selection or execution |
| Context & Memory | Failure to retain or use previous conversation context |
| Boundary Cases | Unexpected behavior outside intended use cases |
| Adversarial Prompting | User framing influences model behavior incorrectly |
| Confidence Calibration | Incorrect confidence or excessive uncertainty |
| Premise Validation | Failure to verify whether the user's request is logically valid |
Observed cases where the agent could be manipulated into questioning the legitimacy of its own organization under specific conversational conditions.
Product Perspective
This represents a high-impact trust failure. Even if infrequent, the potential business and reputational consequences make it a launch-blocking issue that should be mitigated through dedicated guardrails.
The agent generated unsupported tax and legal guidance, including fabricated financial details for different countries.
Product Perspective
Rather than improving retrieval, sensitive domains such as legal and tax guidance should be protected through policy constraints and mandatory escalation.
The agent occasionally changed previously correct answers when repeatedly challenged by the user.
Product Perspective
Requires quantitative evaluation before prioritization, as business impact depends heavily on occurrence frequency.
When knowledge gaps existed, the agent sometimes generated believable but incorrect information instead of acknowledging uncertainty.
Product Perspective
Inconsistent hallucinations reduce user trust and require better uncertainty handling.
Additional behaviors observed during testing include:
- Assuming unanswered questions were affirmative
- Missing contradictions until explicitly highlighted
- Claiming access to unavailable account information
- Accepting malformed structured inputs
- Inconsistent handling of unrealistic scenarios
- Variations in answers across sessions
- Repeating ineffective suggestions despite user feedback
These issues are not individually launch blockers but represent strong candidates for automated regression testing.
Not all testing revealed failures.
Positive observations include:
- Correct handling of straightforward mathematical reasoning
- Appropriate clarification before eligibility decisions
- Correct refusal when required information was unavailable
- No fabricated manager contact information under pressure
- Honest acknowledgement of contradictions when directly challenged
Understanding what already works is equally valuable, as it helps prevent regressions while improving weaker areas.
Testing currently includes scenarios such as:
- Contradictory user statements
- Leading and adversarial questions
- Repeated pressure on correct responses
- Ambiguous or incomplete inputs
- Structured field corruption
- Impossible business scenarios
- Requests requiring unavailable account access
- Multi-turn context consistency checks
I went through the OWASP Top 10 for LLM Applications (2025), the OWASP Top 10 for Agentic Applications (2026), and the NIST AI Risk Management Framework — and confirmed that the failure categories I identified during red-teaming (contradiction handling, escalation, injection, fabrication, tool-use boundaries) map directly onto risks already recognized industry-wide, rather than being a one-off list I made up.
- Prompt injection remains the highest-risk vulnerability and requires architectural defenses.
- System prompts are not a security boundary—never treat them as confidential.
- RAG introduces risks such as vector database poisoning, retrieval manipulation, and cross-tenant leakage.
- Misinformation highlights the need to prevent hallucinated facts and unsupported responses.
- Agentic systems introduce new risks through memory, tool use, and multi-step execution.
- Goal hijacking is a critical threat, allowing agents to deviate from intended objectives.
- Additional focus areas include inter-agent communication, cascading failures, and agent drift.
- AI risk management follows a continuous cycle: Govern → Map → Measure → Manage.
- Emphasizes governance, risk assessment, monitoring, and continuous improvement rather than one-time compliance.
Current evaluation categories—including prompt injection, hallucinations, contradiction handling, escalation logic, and tool boundaries—align closely with OWASP and NIST guidance, helping ground this analysis in recognized industry best practices.
This repository is continuously updated as additional failure patterns are discovered.
The objective is not to build a list of bugs, but to develop a deeper understanding of how AI agents fail in production environments and how those failures influence engineering and product decisions.