A Multi-Agent Framework for Mitigating Dialect Biases in Privacy Policy Question-Answering Systems

Technical Summary: A Multi-Agent Framework for Mitigating Dialect Biases in Privacy Policy Question-Answering Systems

Problem Statement
Privacy policies are critical for informing users about data collection and usage, yet their complexity often renders them inaccessible. While Privacy Policy Question Answering (QA) systems aim to bridge this gap, existing models exhibit significant performance disparities across English dialects. Standard models, typically trained on Standard American English (SAE), struggle with non-standard varieties such as African American Vernacular English (AAVE), Jamaican English, and Aboriginal English. This dialectal bias disproportionately disadvantages marginalized communities, limiting their ability to access and understand their privacy rights. Current approaches to mitigating these biases often require extensive dialect-specific training data or fine-tuning, which is frequently infeasible in sensitive domains like privacy and legal reasoning due to data scarcity and ethical constraints.

Methodology
The authors propose a novel multi-agent framework inspired by human-centered design principles to mitigate dialectal biases without requiring task-specific retraining or large dialectal datasets. The system operates through a structured collaboration between two specialized agents:

1. Dialect Agent: Acting as a linguistic expert, this agent receives the user's query in a specific non-standard dialect. It is prompted with detailed background information regarding the dialect's phonetic, grammatical, lexical, and cultural nuances. Its primary tasks are:

   • Translation: Converting the dialectal query into Standard American English (SAE) while preserving the original intent and cultural context.
   • Validation: Evaluating the final answer generated by the Privacy Agent to ensure it aligns with the user's original dialectal intent, flagging potential misunderstandings caused by dialect-specific phrasing.

2. Privacy Policy Agent: Acting as a domain expert, this agent processes the SAE-translated query against the relevant privacy policy text. It generates an initial answer and rationale based strictly on the policy content.

Iterative Refinement Process:
The framework employs a feedback loop to ensure accuracy. If the Dialect Agent identifies discrepancies between the Privacy Agent's answer and the user's original dialectal intent, it triggers a reconsideration stage. The Privacy Agent then refines its response based on the Dialect Agent's feedback. This loop repeats for a maximum of two iterations to balance performance gains with computational efficiency.

Key Contributions

• Comprehensive Benchmarking: The paper provides an exhaustive benchmark of dialect biases across state-of-the-art Large Language Models (LLMs) including GPT-4o-mini, Llama 3.1, and DeepSeek-R1, applied to PrivacyQA and PolicyQA datasets.
• Novel Multi-Agent Framework: The authors introduce a zero-shot and few-shot capable framework that integrates direct dialect knowledge via prompting rather than model fine-tuning. This approach allows the system to adapt to minority dialects without collecting large amounts of training data.
• Ablation and Error Analysis: The study includes a detailed analysis of the framework's components, demonstrating the specific impact of dialect-specific background information and the frequency of corrective interventions by the Dialect Agent.

Results
Evaluated on the PrivacyQA and PolicyQA datasets using the Multi-VALUE framework to generate dialectal variations, the proposed method yields significant improvements:

• Accuracy Gains: On PrivacyQA, the framework improved GPT-4o-mini's zero-shot accuracy from 0.394 to 0.601. On PolicyQA, accuracy increased from 0.352 to 0.464.
• Bias Reduction: The framework substantially reduced performance disparities across dialects. For GPT-4o-mini on PrivacyQA, the maximum difference in F1 scores between dialects (Max Diff) was reduced from 0.093 to 0.025, representing an 82% reduction in disparity.
• Competitiveness: The multi-agent zero-shot approach often matched or surpassed the performance of few-shot baselines that utilized additional in-context examples, demonstrating the efficacy of the structured collaboration.
• Generalizability: Improvements were observed across all tested models and dialects, including RAAVE, Jamaican, Aboriginal, Welsh, and Southwest England Dialects. Notably, the framework also slightly improved performance on SAE, suggesting the collaborative refinement process benefits all users.

Significance and Claims
The paper claims that its approach offers a scalable, lightweight solution for enhancing fairness in NLP systems, particularly in high-stakes domains like privacy where data collection for fine-tuning is difficult. By leveraging structured prompting and multi-agent collaboration, the framework demonstrates that incorporating dialect and cultural context can significantly improve model equity without additional training data. The authors emphasize that designing NLP systems with an awareness of linguistic diversity is essential for ensuring equitable access to critical information. They suggest that while their method relies on synthetic dialect data and explicit dialect labels, it highlights the potential for "small, well-informed modifications to prompting strategies" to have a meaningful impact on reducing bias in zero-shot settings. Future work is suggested to explore automated dialect detection and the application of this framework to other high-stakes domains such as healthcare and legal AI.