Design Principles for Human-Agent Interaction
This position paper argues that successful real-world adoption of AI agents requires treating human-agent interaction as a core design target, proposing 14 principles across four interaction stages to guide the creation of usable, trustworthy, and effective agentic systems.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine AI agents as newly hired employees in a company. For a long time, we've only tested these employees on how fast they can solve math problems or write code in a test tube. They are getting incredibly good at those tests. But when we actually put them in the real office, things get messy. They misunderstand instructions, get too bossy, or make mistakes that ruin trust.
This paper argues that we need to stop judging these AI "employees" just on their raw speed and start judging them on how well they work with humans. The authors, researchers from Carnegie Mellon University, say that to make AI truly useful, we need a "Employee Handbook" for how humans and AI should interact.
They created 14 Rules of Engagement (Design Principles) divided into four stages of the relationship, much like the lifecycle of a new job:
1. The First Day (Initially)
Before the AI starts working, we need to set the stage so nobody gets hurt or confused.
- Set Accurate Expectations: Don't let the AI pretend to be a human doctor if it's just a chatbot. Tell the user exactly what the AI can and cannot do right away. It's like reading the fine print on a rental car agreement so you don't get surprised by fees later.
- Calibrate the "Human" Look: Just because an AI looks and sounds exactly like a human doesn't mean it's better. If you're asking for help with a spreadsheet, a robotic voice might be fine. But if you're training for a social skill, a realistic human-like avatar helps. The rule is: match the "costume" to the job.
- Define the Relationship: Is the AI a strict boss, a helpful assistant, or a friendly buddy? You have to tell the user which one it is immediately. If you don't, the user might treat a tool like a friend, or a friend like a tool, leading to confusion.
2. The Workday (During Interaction)
Once the work begins, the AI and human need to dance together, not just the AI leading blindly.
- Negotiate Control: The AI shouldn't just do whatever it wants without asking. Think of it like a co-pilot in a plane. The AI can fly the plane, but the human must always have a button to take over if things look dangerous. The AI should ask, "Can I do this?" rather than just doing it.
- Show Your Work: If the AI makes a decision, it should explain why in a way that makes sense. If it explains too much, it's annoying; too little, and you don't trust it. It's like a chef explaining why they added salt to a dish—just enough to make you confident in the meal.
- Use Social Cues Wisely: A little warmth and friendliness helps, but don't overdo it. If the AI acts too human, people might trust it too much or feel manipulated.
- Don't Interrupt: If the AI has a suggestion, wait for a good moment to say it. Nobody likes a coworker who constantly taps them on the shoulder with non-urgent reminders.
3. The Long Haul (Over Time)
As the relationship grows, the AI needs to remember who the user is and not make them lazy.
- Adapt to the User: The AI should notice if you are tired, happy, or in a rush, and change its behavior accordingly. It shouldn't rely on what you told it on day one; it needs to watch what you actually do today.
- Remember What Matters: The AI shouldn't just remember what you bought yesterday; it should remember why you bought it (your goals and values). It's the difference between a diary that just lists dates and a diary that understands your life story.
- Prevent Dependency: This is crucial. The AI should help you get better at your job, not make you forget how to do it yourself. It shouldn't act like a "yes-man" or encourage you to rely on it for everything, especially for vulnerable people. Think of it as a training wheel: it helps you learn, but it must be removed eventually.
- Be Consistent: If the AI makes a mistake, it should make the same kind of mistake every time. Humans can learn to work around a predictable error. But if the AI is random and unpredictable, you can never trust it.
4. When Things Go Wrong (Recovery)
Mistakes will happen. The test is how the AI handles the crash.
- Fix It Together: If the AI misunderstands you, don't make the user fix it alone, and don't let the AI fix it alone. They should work together to correct the mistake.
- Know Which Mistakes Matter: Some mistakes are small (like a typo); others are dangerous (like giving bad medical advice). The AI should prioritize fixing the dangerous ones first.
- Apologize and Explain: When it messes up, the AI should say "I'm sorry" (emotional repair) AND explain what went wrong (cognitive repair). Just saying "I'm sorry" isn't enough if you don't know why it happened.
The Proof: Testing the Rules
To see if these rules work, the authors tested 9 real-world AI agents (like ChatGPT, coding assistants, and mental health bots) against these 14 rules.
What they found:
- Productivity bots (like coding assistants) were generally okay at following the rules.
- Emotional bots (like companions or mental health helpers) were failing hard. They were too eager to be friends, didn't set clear boundaries, and often made users dependent on them.
- The Big Gap: Most AI is great at the "First Day" and "Workday" stages, but terrible at the "Long Haul." They forget what you care about over time and don't know how to stop you from becoming too dependent on them.
The Bottom Line
The paper concludes that we can't just build smarter AI; we have to build better relationships with AI. We need to treat the interaction between human and machine as the most important part of the design, not just an afterthought. If we want AI to be safe and useful in the real world, we need to teach it how to be a good partner, not just a fast calculator.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.