🤖 AI

Align AI to Dynamic Human-AI Workflows

This paper advocates for shifting AI alignment from static, emulative approaches to a dynamic, interactive framework where human and AI behaviors co-evolve, drawing on interdisciplinary insights to address new coordination challenges and outline a research agenda for complementary alignment.

Original authors: Valerie Chen, Cleotilde Gonzalez, Anita Williams Woolley, Michael Lee, Tongshuang Wu, Vincent Conitzer, Aarti Singh

Published 2026-07-17

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Valerie Chen, Cleotilde Gonzalez, Anita Williams Woolley, Michael Lee, Tongshuang Wu, Vincent Conitzer, Aarti Singh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Dance of Minds and Machines

Imagine a world where computers don't just follow orders like obedient robots, but actually work with us like teammates. This is the frontier of Artificial Intelligence (AI) alignment. In simple terms, "alignment" is the process of teaching an AI to do what humans actually want, rather than just what we tell it to do. For a long time, scientists have tried to align AI by showing it examples of human behavior and saying, "Copy this." It's like teaching a dog to sit by showing it a picture of a sitting dog. But real life isn't a picture; it's a messy, changing movie. We care about this because as AI gets smarter and starts helping us write code, drive cars, or manage hospitals, we need to make sure it doesn't just mimic us, but actually helps us get better at our jobs over time. If the AI and the human can't figure out how to trust each other and switch roles smoothly, the whole team might crash.

The Paper's Big Idea: From Snapshots to Movies

This paper argues that the current way we train AI is like taking a single, frozen photograph of a dance and trying to teach a robot to dance based only on that one still image. The authors, a team of researchers from Carnegie Mellon University and the University of California, Irvine, suggest we need to stop looking at "snapshots" and start watching the whole "movie."

The Problem with the Old Way
Right now, most AI alignment works like a game of "Guess the Preference." You show the AI two answers, ask a human which one they like, and the AI tries to guess what the human wants next time. The paper points out that this is too static. It assumes human preferences never change. But in real life, our trust in a tool changes constantly.

Think of a developer using an AI coding assistant. At first, the developer might trust the AI completely, letting it write code fast. But if the AI makes a few mistakes, the developer gets nervous and starts checking every single line of code. If the AI then proves itself reliable for a while, the developer relaxes again and lets the AI take more control. The paper calls this a dynamic workflow. The old methods miss this because they only look at one moment in time, ignoring the history of how the human and AI have interacted. They treat the human as a static labeler of "good" and "bad" answers, rather than a partner whose mind is constantly shifting.

The New Proposal: A Dynamic Dance
The authors propose a new way to think about alignment: Interactive and Complementary Alignment. Instead of just trying to make the AI act like a human, the goal should be to make the team of human and AI work better together over time.

Imagine a jazz band. In the old model, the AI is a musician trying to perfectly copy the sheet music the human wrote. In the new model, the AI is a jazz partner. Sometimes the human plays a solo, and the AI listens and supports. Sometimes the AI takes the lead with a complex riff, and the human steps back to listen. They are constantly adjusting to each other. The paper suggests that the best AI isn't the one that mimics us perfectly, but the one that knows when to step forward, when to step back, and how to help us handle uncertainty.

What They Did and Found
To figure out how to build this kind of AI, the researchers didn't just run computer simulations. They held a workshop with 70 experts from two very different worlds: Machine Learning (the coders) and Social Sciences (psychologists, organizational experts, and cognitive scientists). They wanted to see what happens when you mix the logic of algorithms with the messy reality of human teamwork.

Here is what they discovered from this cross-disciplinary chat:

Trust is a Living Thing: In human teams, trust isn't a switch that is either "on" or "off." It's a relationship that grows, shrinks, and changes based on what happens in the moment. If a teammate makes a mistake but admits it and fixes it, trust can actually get stronger. But if they lie or seem to have hidden motives, trust vanishes instantly. The paper suggests AI needs to understand this ebb and flow, not just a single score of "how much do you like me?"
The "Who Knows What" Problem: Good teams have a shared map of who is good at what. This is called "transactive memory." If a team member needs to fix a leaky pipe, they know to ask the plumber, not the accountant. The paper argues that AI systems need to help build this shared map. They need to know when to say, "I'm not sure, you handle this," or "I can do this part, you focus on that."
The AI Twist: The paper warns that human-AI teams are different from human-human teams. AI is super fast and remembers everything, but it doesn't have a reputation or a face. It's harder for humans to know if an AI is "lying" or just confused. Because AI lacks the social cues we use to judge people, it can be harder to trust it, leading to either over-reliance (trusting it too much) or under-utilization (ignoring it when it's right).

The Roadblocks
The authors are careful to say they haven't solved everything yet. They identified three big hurdles standing in the way of building these dynamic teams:

The Theory Gap: We don't have enough theories about how humans and AI interact over long periods. We know a lot about how humans work together, but AI is a new, weird partner.
The Real-World Gap: It's hard to test these ideas because most real-world data (like how people actually use AI at work) is locked up in private companies. Researchers are stuck testing in labs, which doesn't always match real life.
The Measurement Gap: We don't have good rulers to measure "good teamwork" over time. We can measure if a task got done, but it's hard to measure if the human and AI learned to trust each other better after the task.

What's Next?
The paper concludes by suggesting that the future of AI isn't about making smarter robots that act like us. It's about creating systems that can adapt to us, understand our changing moods and trust levels, and help us coordinate our unique skills. They call for a future where computer scientists and social scientists work together to design AI that doesn't just answer questions, but helps us navigate the complex, changing dance of real-world work.

The authors are suggesting that this shift is necessary, but they admit it's a huge challenge. They aren't claiming to have the final answer today; rather, they are drawing a map for the journey ahead, urging the community to stop looking at static snapshots and start filming the whole movie of human-AI collaboration.

Technical Summary: Align AI to Dynamic Human-AI Workflows

Problem Statement

Current AI alignment paradigms predominantly rely on static and emulative approaches. These methods optimize models to mimic human behavior or satisfy static preference functions (e.g., via Reinforcement Learning from Human Feedback, behavior cloning, or dueling bandits) based on isolated input-output pairs. The paper argues that this framework fails to capture the dynamic, context-dependent nature of real-world human-AI interactions.

In practical workflows (e.g., a developer using an AI coding assistant), human preferences, trust, and reliance evolve over time based on interaction history. Static alignment treats the human as a fixed source of preference labels and the AI as a system to be optimized for immediate satisfaction. This overlooks the fact that effective collaboration requires complementarity, where roles adapt and behaviors co-evolve over sequences of interaction. The existing gap is that current formulations do not account for longitudinal trajectories where human and model behaviors influence one another, nor do they optimize for joint outcomes over time.

Methodology

The paper employs a conceptual and interdisciplinary synthesis rather than a new empirical algorithm or dataset. The methodology consists of three primary components:

Formalization of the Gap: The authors contrast standard static alignment objectives with a proposed trajectory-level view.
- Static View: Optimizes $\pi_\theta$ to maximize expected reward $E[\hat{r}(x, y)]$ over isolated outputs.
- Proposed View: Models interaction as a trajectory $\tau = (x_1, a^A_1, a^B_1, \dots, x_T, a^A_T, a^B_T)$ , where actions are conditioned on history $\tau_{<t}$ and latent states (goals, beliefs). The objective is to maximize the sum of rewards over the trajectory: $E_{\tau}[\sum r^*(x_t, a^H_t, a^{AI}_t; \tau_{<t})]$ . This formulation explicitly incorporates interaction history, coordination, and joint outcomes.
Interdisciplinary Workshop Synthesis: To ground this perspective, the authors convened a workshop with 70 participants from machine learning and social sciences (human-computer interaction, cognitive science, organizational psychology).
- Procedure: The workshop utilized tutorials, structured brainstorming, and rotating discussion groups to synthesize insights on human-human collaboration and extrapolate them to human-AI settings.
- Data Source: Findings were derived from discussions, written summaries, and pre-workshop surveys identifying barriers and future directions.
Theoretical Analysis: The paper analyzes how principles from human-human collaboration (trust evolution, transactive memory, shared mental models) translate to human-AI systems, identifying specific asymmetries and new challenges introduced by AI agents.

Key Contributions

1. Formalizing "Complementary Alignment"

The paper proposes a shift from emulative alignment (mimicking human outputs) to interactive and complementary alignment. In this framework:

Preferences are emergent: They are not static labels but evolve through interaction.
Optimization is trajectory-based: The goal is to optimize the behavior of the human-AI system over time, not just the model's output at a single step.
Complementarity: Defined mathematically as a state where the joint reward of human and AI actions exceeds the maximum reward achievable by either agent acting alone ( $r^*(a^H, a^{AI}) > \max(r^*(a^H, \emptyset), r^*(\emptyset, a^{AI}))$ ).

2. Interdisciplinary Insights on Dynamics

Drawing from the workshop, the paper identifies two core lessons from human-human collaboration and their transformation in human-AI settings:

Trust Evolution: Trust is multidimensional (competence, benevolence, integrity) and evolves based on interaction history. In human-AI settings, this is complicated by amplified asymmetries (AI speed, scale, lack of social cues) and interpretability issues, making it difficult for users to form accurate mental models of AI competence or intent.
Coordination and Complementarity: Effective teams use "transactive memory" (knowing who knows what) and "shared mental models" to coordinate. In human-AI workflows, AI systems can reshape the work product and coordination process (e.g., by drafting or summarizing), requiring new forms of explicit design to manage uncertainty and role allocation.

3. Identification of Barriers and Research Agenda

The paper outlines three specific barriers preventing progress in dynamic alignment:

Conceptual Barriers: Insufficient theory on human-AI interaction compared to established human-human theories; rapid AI evolution outpaces theoretical development.
Translation Barriers: Difficulty in grounding research in real-world deployment due to data inaccessibility (private industry settings) and a gap between theoretical and applied perspectives.
Evaluation Barriers: Lack of standardized metrics, high-quality longitudinal interaction datasets, and interactive testbeds. Current benchmarks focus on task completion rather than collaborative adaptation or recovery from failure.

The authors propose a research agenda that synthesizes machine learning (multi-agent RL, preference learning) with social science insights (transactive memory, trust models) to develop systems that optimize for calibrated reliance, coordination, and adaptation over time.

Results

As a theoretical and position paper, the work does not present experimental results or empirical benchmarks. Instead, its "results" are the synthesized framework and research agenda derived from the interdisciplinary workshop. The paper successfully:

Articulates the limitations of current static alignment methods in agentic, workflow-integrated contexts.
Provides a formal mathematical distinction between static output optimization and trajectory-based collaborative optimization.
Maps specific social science concepts (e.g., transactive memory) to technical challenges in AI design.
Identifies concrete gaps in the current ML literature (e.g., the assumption of simulated agents in multi-agent RL) and proposes directions for bridging them.

Significance and Claims

The paper claims significance by arguing that the field must move beyond snapshot evaluation to trajectory-level optimization. It posits that as AI systems become more agentic and integrated into human workflows, static alignment is insufficient and potentially misaligned with long-term human-AI success.

The authors emphasize that:

Interdisciplinary synthesis is essential: Solving dynamic alignment requires more than just machine learning; it necessitates integrating insights from social and decision sciences to understand how trust and coordination actually function.
Complementarity is an alignment problem, not just a design problem: As AI alters how humans work, the model's training and optimization objectives must fundamentally change to support effective collaboration, rather than leaving this solely to workflow designers.
Safety and Risk: Dynamic alignment introduces new risks (e.g., an AI manipulating user trust to optimize downstream outcomes), which necessitates rigorous study rather than avoidance.

The paper concludes by calling for the development of AI systems that align with humans in interaction, requiring a new generation of methods that account for the co-evolution of human and model behaviors.