When Agents Misbehave: UX Failure Modes and Design Patterns for Human-Agent Coordination

Imagine you ask an AI agent to reorganize your project files. It does - but it interpreted "reorganize" differently than you meant, and now three months of work are sorted in a way that makes no sense to you. There's no undo button. There's no explanation. There's no way back.

This is the problem I set out to solve. As AI shifts from reactive chat to autonomous action, a critical gap has emerged: when agents misinterpret intent, hallucinate, or execute the wrong strategy, users have no natural recovery path. The result isn't just frustration - it's a fundamental breakdown in trust that makes people abandon the system entirely.

I led the research and design of recovery and repair frameworks for four core agentic failure modes: wrong intent, limited ability, hallucination, and no answer. The design patterns that emerged are being used in AWS SageMaker and under review for academic publication - and are being considered as guidance for the broader AWS organization.

A note on confidentiality: this is ongoing proprietary work. Some specific patterns and study results are abstracted until the associated research paper is published.

Project overview

As AI shifts from reactive chat to agentic systems that act semi-independently, a critical gap has emerged: the "Black Box" problem. When agents misinterpret intent or execute unintended strategies, the lack of recovery mechanisms leads to a breakdown in user trust. We are focussing on four main error types - wrong intent, limited ability, hallucinations and no answer. Since Agentic AI is an emerging frontier with no established industry playbooks, this project represents a foundational strategic shift for the organization. The interaction models and design patterns (like, Transparency, Prevention and Fast Recovery, Behavioral and Conversational patterns) we have established through this project are already being deployed within AWS SageMaker, and currently being reviewed for guidance for the entire AWS organization.

The challenge

This project investigates how users interact with an AI agent after it makes a mistake - whether that’s misinterpreting an instruction, executing an unintended task, or choosing the wrong strategy. The focus is on designing and prototyping repair and recovery mechanisms that feel natural and empower users to regain control. The goal is to transform failure modes from dead-ends into "teachable moments" that empower users to regain control without abandoning the system.

Role and scope

UX Designer, along with a UX researcher and a Principal design leader

  • I drove the end-to-end research strategy, from initial desk research to designing patterns for prototyping to conducting a research study with 15 AI engineers, product managers and designers, and the patterns that emerged from it.

  • Collaborated with engineers to develop production level prototypes, in addition to 1 UX researcher and a Principal design leader

Impact

In a world of autonomous systems, Recovery is the Product. My work didn't just solve for errors; it defined a new discipline of human-agent coordination that ensures that as our systems become more independent, they remain fundamentally human-centric and trustworthy. The design patterns that emerged from this work are already being implemented within AWS Sagemaker and are being reviewed for publication in academic journals.

The Process

Categorizing Failure Modes

I worked with engineers to build a functional prototype to test four specific forms of errors in agentic systems

  1. Wrong Intent: The agent misunderstands the user intent.

  2. Limited Cognitive or Computational Ability: The agent attempts a task it is not equipped to finish.

  3. Hallucinations: Plausible sounding but factually incorrect answers.

  4. No answer ( improper access or resource does not exist): The user asks something that the agent doesn’t have permission to execute or that the agent is not aware of

Design thinking process

Steps to arrive at design patterns that were then prototyped and tested

  1. Design thinking workshop : Brainstormed design interaction patterns that help users recover from AI mistakes intuitively and quickly, while boosting their confidence and helping them feel in control when the AI system makes mistakes.

  2. UX patterns and use cases: Defined high level patterns like salience, reasoning transparency, escalation, audit trails, etc. to ground the prototype and work through sample use cases for all 4 error types and their expected behavior depending on these pattern levers.

  3. Collaboration with engineering: Worked with engineering to flesh out these patterns along with the technical feasibility of simulating these in the prototype for further study with AI engineers, designers and product managers.

Research phase

We examined how such misbehaviors are experienced through interviews with 15 senior engineers, product managers, and UX designers. To scaffold reflection, we built a functional agent that simulated the above four forms of misbehavior, prompting participants to surface analogous episodes from their own practice. Research questions we were looking to answer:

  • How can users reverse or undo agent actions after an error?

  • What feedback loops help the agent learn from mistakes?

  • How can the system support both hands-on correction and passive guidance?

  • What UI patterns restore user confidence after AI error?

  • How do different users prefer to recover from AI mistakes?

From research to interaction patterns

Our findings show how misbehavior leads to workarounds, renegotiation of roles, and shifting notions of trust and control, while highlighting the need for agents to surface human-like behaviors such as uncertainty and honest admission of limits. Based on the analysis, we established new design patterns across Transparency, Prevention and Fast Recovery, Behavioral and Conversational patterns that address these failure modes in human–agent coordination.