Technical Summary: It's Complicated: On the Design and Evaluation of AI-Powered AAC Interfaces

Problem Statement

The integration of Artificial Intelligence (AI) into Augmentative and Alternative Communication (AAC) systems presents a significant evaluation challenge. Current evaluation paradigms rely heavily on quantitative technical benchmarks (e.g., word error rate, words per minute) common in machine learning and natural language processing. However, these metrics often fail to capture the multifaceted, intersectional, and dynamic nature of human communication.

The paper argues that treating AAC users as a static, singular "standard user" leads to technoableism, where systems are optimized for hypothetical efficiency rather than the lived reality of users with diverse identities, physical abilities, and communication contexts. This reductionist approach risks pathologizing disability and ignoring the "misfit" between a user's body-mind and their environment. Consequently, a system may appear technically successful while failing to support the user's agency, identity, or social participation.

Methodology and Approach

This paper does not present a new experimental dataset or a specific AI model implementation. Instead, it employs a conceptual and analytical framework to critique existing evaluation methods and propose a more robust, pluralistic approach.

The methodology involves:

1. Intersectional Analysis: Applying the lens of intersectionality to AAC design, recognizing that users exist at the intersection of race, gender, disability status, and societal roles.
2. Problem Space Exploration: Analyzing six critical domains where AI can augment AAC, examining the specific design considerations, potential AI interventions, and the limitations of current metrics within each space.
3. Proposed Evaluation Framework: Synthesizing technical performance metrics with human-centered design research methods (e.g., interviews, diary studies, usability testing) to create a multidimensional evaluation strategy.

Key Contributions

The paper identifies six critical design considerations for AI-powered AAC interfaces and proposes specific evaluation strategies for each:

1. Communication Speed and Accuracy

• Challenge: Traditional metrics like Words Per Minute (WPM) and Word Error Rate (WER) do not account for the trade-off between speed and the user's intent, nor do they capture semantic accuracy when exact phrasing is not required.
• AI Potential: Large Language Models (LLMs) can offer "big swing" predictions (entire sentences) and leverage contextual clues (environment, partner input) to improve prediction.
• Proposed Evaluation: A multidimensional approach including:
  • Semantic/Linguistic Similarity (Offline): Using LLM judges and semantic embeddings to assess intent preservation rather than verbatim matching.
  • User-Mediated Assessment (Online): Tracking user correction rates and acceptance of "close-enough" predictions to measure pragmatic trade-offs.
  • Functional Success (Task-Based): Evaluating whether the communication achieved a social goal (e.g., requesting an action) rather than just text entry accuracy.

2. Physical and Mental Effort

• Challenge: Users with physical disabilities often face fatigue, eye strain, or high cognitive loads when navigating interfaces. High-effort access methods (e.g., single-switch scanning) can be exacerbated by system errors.
• AI Potential: AI can detect physiological signs of fatigue (via EEG or interaction patterns) and dynamically adapt the interface (e.g., increasing prediction size or accuracy at the cost of latency) to reduce effort.
• Proposed Evaluation:
  • Physical Effort: Measuring motor actions required (e.g., switch activations) or using the CARE Efficiency Score.
  • Mental Effort: Utilizing self-reported tools like the NASA Task Load Index or sensor-based techniques (eye-tracking, EEG) to assess cognitive load.

3. Sounding Like You Want to Sound (Identity and Voice)

• Challenge: Speech-generating devices often fail to replicate natural speech nuances (accents, cadence, tone), leading to misrepresentation and loss of social presence.
• AI Potential: AI can synthesize voices based on banked data or blend samples to create specific accents. It can also adjust prosody based on tone indicators and context.
• Proposed Evaluation: Usability testing and diary studies to assess user satisfaction with tone indicators. Rating scales should measure "self-identification" (does the voice feel like an extension of the self?) and partner satisfaction regarding the conveyance of emotion and social standing.

4. Code- and Context-Switching

• Challenge: Users switch between languages, dialects, and formality levels based on context (e.g., chatting with a friend vs. speaking to a doctor). Current systems offer limited support for these shifts.
• AI Potential: AI can detect the communication partner and context to adjust prediction styles (casual vs. formal) and handle multilingual code-switching.
• Proposed Evaluation: Qualitative methods (interviews, ethnography) to map communication styles, followed by usability testing and the use of the Communicative Participation Item Bank (CPIB) to measure success in adapting to different contexts.

5. Taking Part Fluidly in a Conversation (Turn-Taking)

• Challenge: The latency of AAC systems often causes users to miss turn-taking opportunities or struggle with backchanneling (e.g., "uh-huh"), leading to social exclusion.
• AI Potential: AI can predict backchanneling needs and generate "take-the-floor" scripts to help users enter conversations spontaneously.
• Proposed Evaluation: Measuring latency to respond and floor-taking success rates quantitatively, complemented by partner interviews to assess if the AI's timing feels organic or uncanny, ensuring the user retains agency in self-presentation.

6. Short- and Long-Term Needs Changes

• Challenge: Communication needs fluctuate due to fatigue, medication, or progressive conditions (e.g., ALS, autism shutdowns). Static systems become unusable as needs change.
• AI Potential: Adaptive interfaces that detect user state (e.g., shutdown) and switch between detailed controls and simplified presets, or adjust prediction granularity based on fatigue levels.
• Proposed Evaluation: A longitudinal approach using diary studies where users report on system adaptations over time, combined with technical metrics on the system's ability to detect needs changes.

Results and Claims

The paper does not present empirical results from a new system but rather synthesizes existing literature to argue for a paradigm shift in evaluation.

• Claim: Technical metrics alone are insufficient and potentially harmful if they ignore intersectional nuances.
• Claim: A "pluralistic" set of evaluation methods is required to capture the full story of user experience.
• Claim: AI features must be evaluated not just on efficiency, but on their ability to support agency, identity, and social participation.
• Claim: The "Three Domains Framework" (Device Design, Wider Picture, Personal Context) should guide evaluation to prevent epistemic injustice.

Significance

The paper asserts that the significance of this work lies in moving the field away from technoableism and toward human-centered, participatory design. It highlights that:

1. Evaluation is Political: How we measure success determines what gets built. Prioritizing speed over identity can devalue the user's sense of self.
2. Complexity is Inherent: Researching humans in AAC is complicated; users do not fit into neat boxes. Evaluation methods must reflect this complexity.
3. Participatory Necessity: Any inclusion of AI in AAC must be done alongside AAC users. The paper calls for a shift where what is measured is guided by what AAC users decide is important, rather than what is convenient for technical benchmarking.

The authors conclude that technical performance metrics should be viewed as only "one star in the constellation of evaluation," necessary but insufficient without the broader context of human-centered data.