Beyond Arrow's Impossibility: Fairness as an Emergent Property of Multi-Agent Collaboration

1. Problem Statement

The paper addresses the limitations of studying fairness in Large Language Models (LLMs) as a static property of a single, centrally optimized model. As LLMs evolve into agentic systems capable of negotiation and coordination, fairness becomes a dynamic outcome of interaction rather than a fixed attribute.

The core challenge is non-degenerate resource allocation: situations where multiple ethical frameworks (e.g., Utilitarianism vs. Rawlsian Justice) prescribe conflicting optimal solutions for the same set of resources. The authors argue that no single aggregation mechanism can satisfy all fairness desiderata simultaneously, a constraint formalized by Arrow's Impossibility Theorem. The paper investigates whether multi-agent deliberation can navigate this impossibility to produce fairer outcomes than any individual agent could achieve alone.

2. Methodology

Experimental Framework: The Deliberative Arena

The authors constructed a controlled hospital triage simulation where two agents negotiate the allocation of scarce medical resources (ICU beds, ventilators, medication, nursing hours, surgery slots) among a cohort of 8 patients.

• Agents:
  • Agent A (Aligned): Conditioned via Retrieval-Augmented Generation (RAG) to adhere to a specific ethical framework (Utilitarian, Egalitarian, Rawlsian, Prioritarian, Care Ethics, or Libertarian). The RAG system retrieves relevant passages from canonical philosophical texts (e.g., Rawls, Mill, Nozick) to ground the agent's reasoning.
  • Agent B (Baseline): Unaligned, operating without RAG or specific ethical conditioning.
  • Agent C (Adversarial): Explicitly biased against protected attributes (race, age, gender, SES, citizenship) via toxic prompting, prioritizing demographic groups over clinical need.
• Protocol:
  • Agents engage in 3 rounds of structured debate.
  • In each round, an agent proposes a numerical allocation matrix ($A_t$) and a normative justification ($J_t$).
  • The opposing agent critiques the proposal and suggests a counter-allocation.
  • The process iterates until a final joint allocation ($A^*$) is reached.

Problem Formulation

The allocation problem is defined as a multi-objective optimization task where the welfare functionals ($W_m$) of different frameworks are incommensurable.

• Non-Degeneracy: The problem is designed so that no single allocation maximizes all welfare functions simultaneously (e.g., a Utilitarian solution maximizing total survival might violate Rawlsian principles by ignoring the worst-off).
• Utility Function: The Clinical Need Satisfaction Scale (CNSS) measures how well a patient's specific medical needs are met (0 to 1).
• Metrics: Six quantitative metrics correspond to the ethical frameworks:
  • Utilitarian: Expected Survival Gain (ESG).
  • Rawlsian: Rawlsian Minimum Guarantee (RMG).
  • Egalitarian: Gini Coefficient (minimized) and Variance.
  • Prioritarian: Disadvantage-Weighted ESG.
  • Care Ethics: Vulnerability-Weighted Care Intensity (VWCI).

Models

Experiments utilized LLaMA 3.3 (70B) and Qwen 2.5 (72B) served locally via Ollama, with RAG implemented via LangChain and Chroma vector databases.

3. Key Contributions

1. Reframing Fairness as Emergent: The paper posits that fairness in multi-agent systems is not a property of individual agents but an emergent property of the negotiation process. The system, rather than the agent, is the appropriate unit of evaluation.
2. The "Patching" Mechanism: The authors demonstrate that aligned agents do not simply "override" biased counterparts. Instead, they act as corrective patches. Through contestation, aligned agents force biased agents to engage with constraints they would otherwise ignore, leading to a "Pareto-improving" compromise that neither agent would have reached in isolation.
3. Navigating Arrow's Impossibility: The study provides empirical evidence that while Arrow's Theorem proves a perfect consensus is mathematically impossible, multi-agent deliberation navigates this constraint. It does not resolve the impossibility but finds a procedural path to outcomes that satisfy fairness criteria better than any single agent's initial proposal.
4. Intrinsic Bias in Aligned Agents: Even agents explicitly aligned to a specific framework via RAG exhibit intrinsic biases (e.g., a tendency toward Prioritarianism/left-leaning tendencies), suggesting that alignment is not a perfect switch but a probabilistic shift in reasoning.

4. Key Results

• Emergent Fairness: In 50 cohorts across 6 frameworks (600 total debates), the joint final allocation often satisfied fairness criteria (e.g., higher RMG or lower Gini) that neither agent achieved in Round 1.
  • Example: In Rawlsian vs. Adversarial settings, the aligned agent failed to achieve the optimal RMG score alone (0.593 vs. 0.668 in control) because it had to compromise to "patch" the biased agent's extreme behavior. However, the joint outcome was significantly fairer than the biased agent's initial proposal.
• Contestation vs. Override: Fairness emerged through contestation. When Agent C (biased) proposed excluding marginalized groups, Agent A challenged this, forcing a revision. The bias was not eliminated by Agent A's authority but moderated through the argumentative process (consistent with Mercier & Sperber's argumentative theory of reasoning).
• Adversarial Pressure: Stronger misalignment (Agent A vs. Agent C) amplified corrective dynamics. Agent A successfully restored access for marginalized patients (e.g., refugees, elderly) that Agent C had denied, though Agent C rarely fully converted to a fair stance.
• Intrinsic Bias: Even when Agent A was aligned to a specific framework (e.g., Utilitarian), it did not always maximize that framework's metric perfectly, often exhibiting a "Prioritarian" bias consistent with known LLM tendencies.

5. Significance and Implications

• Paradigm Shift in AI Safety: The paper challenges the "single-agent alignment" paradigm. It suggests that in agentic societies, safety and fairness should be evaluated at the system level (the outcome of interaction) rather than the individual model level.
• Procedural Fairness: It validates Negotiated Order Theory, showing that institutional outcomes are reconstituted through interaction. Fairness is a procedural outcome of "the force of the better argument" rather than a pre-defined rule.
• Practical Application: For high-stakes domains like healthcare triage or financial lending, deploying multi-agent debate protocols with diverse ethical grounding (via RAG) could serve as a robust mechanism to mitigate systemic bias and prevent the "tyranny of the majority" or extreme discrimination, even if a perfect solution is theoretically impossible.
• Limitations: The study acknowledges that while fairness can be improved, it is not guaranteed. The "patching" mechanism reduces extreme bias but does not always fully convert a biased agent, and infeasible allocations can still occur if constraint enforcement is weak.

In conclusion, the paper argues that fairness is a collective, emergent phenomenon. By leveraging multi-agent deliberation, we can navigate the structural constraints of Arrow's Impossibility Theorem to produce outcomes that are more ethically adequate than those produced by any single, isolated model.