The Evaluation–Action Gap in Digital Accessibility Research: A Systematic Review of Automation Bias, Sectoral Concentration, and User Exclusion

Technical Summary: The Evaluation–Action Gap in Digital Accessibility Research

Problem Statement
Despite a decade of extensive scholarly attention and a growing volume of accessibility evaluations, digital services remain plagued by persistent barriers that limit equitable access for people with disabilities. This study addresses a critical paradox: while research output has surged, observable improvements in real-world accessibility have not kept pace. The paper identifies an "evaluation–action gap," characterizing the field as "diagnostically rich but intervention-poor." Current research tends to repeatedly document known failures without translating findings into actionable improvements, user-centered interventions, or structural changes. The study seeks to understand the methodological patterns, sectoral concentrations, and limitations that sustain this gap.

Methodology
The study employs a systematic mixed-methods literature review of 104 empirical studies published between 2014 and 2025. The sample was drawn from multidisciplinary databases (including Scopus, Web of Science, and IEEE Xplore) and filtered through strict inclusion criteria: studies must be empirical, published in English, focus on web accessibility (websites, web pages, or web-based systems), and report findings on barriers or compliance.

• Data Extraction and Coding: Data were manually extracted and coded using Microsoft Excel to ensure accessibility for the researcher. Variables included publication year, geographic location, sector, sample type, evaluation methods, tools used, WCAG versions, and specific accessibility barriers.
• Analysis: The study utilized a structured normalization and categorical synthesis approach rather than statistical meta-analysis, due to the heterogeneity of study designs. Python-based computational methods were used to calculate frequencies and distributions, while qualitative thematic interpretation was applied to recurring barriers and recommendations.
• Quality Appraisal: Studies were assessed for methodological clarity, transparency, and reporting of WCAG versions. Studies with inaccessible formats (e.g., untagged PDFs) were excluded, reflecting the very barriers the research critiques.

Key Findings

• Methodological Bias and Automation: Automated evaluation tools dominate the literature, accounting for 49.9% of all reported methods. Three tools—WAVE, TAW, and AChecker—constitute over two-thirds of all automated testing instances. In contrast, user-centered methods (e.g., usability testing, interviews) represent less than 10% of approaches.
• Sectoral and Geographic Concentration: Research is heavily concentrated in the education sector (40.8%) and government websites (25.2%), with commercial sectors like e-commerce and banking minimally represented. Geographically, the literature is skewed toward the Global North, with the United States, Saudi Arabia, and India leading single-country studies, while many regions remain underrepresented.
• Persistent Barriers: Despite the volume of research, fundamental accessibility barriers remain unchanged over the decade. The most prevalent issues are missing text alternatives (79.6% of studies), structural/semantic markup issues (53.4%), and color/contrast failures (53.4%).
• WCAG Compliance: WCAG 2.0 remains the dominant standard (67.96% of studies), with a lag in the adoption of newer versions (WCAG 2.1 and 2.2).
• Recommendations vs. Action: While 55.3% of recommendations focus on technical and design solutions, and 46.6% call for user involvement, the literature rarely translates these into concrete, context-sensitive action plans. The field is characterized by a reliance on "methodological convenience" (automation) over "epistemic adequacy" (user experience).

Key Contributions
The paper makes three primary contributions to the field of digital accessibility:

1. Empirical Mapping: It provides a comprehensive mapping of accessibility evaluations across sectors, identifying common barriers and domain-specific differences.
2. Structural Critique: It highlights structural imbalances in the literature, specifically the marginalization of user-centered approaches and the over-concentration of research in specific sectors and regions.
3. Conceptual Framework: It advances the critical interpretation of the field by framing contemporary research through the lens of the "evaluation–action gap." This concept posits that the field is trapped in a diagnostic loop, prioritizing the documentation of failures over the development of inclusive, actionable solutions.

Significance and Claims
The paper argues that the current trajectory of accessibility research is unsustainable. The reliance on automated tools and the exclusion of disabled users from the research process have created an epistemic narrowing where research is driven by what is technically measurable rather than what is socially consequential.

The study concludes that increasing the volume of compliance-based evaluations will not, in itself, lead to improved accessibility outcomes. Instead, the field requires a fundamental reorientation toward participatory methods, cross-sector collaboration, and solution-oriented research. The author asserts that without shifting from "documentation toward inclusive, actionable change," web accessibility research risks remaining an object of continuous diagnosis rather than a driver of meaningful social and technological inclusion. The paper also notes the irony that conducting such a review presented significant barriers for a visually impaired researcher, underscoring the need for accessible academic publishing standards.