💻 computer science

TactileEval: A Step Towards Automated Fine-Grained Evaluation and Editing of Tactile Graphics

This paper introduces TactileEval, a three-stage automated pipeline that establishes a five-category quality taxonomy for tactile graphics, validates it with over 14,000 annotations to achieve high-accuracy classification, and leverages these insights to drive a ViT-guided system for targeted image editing and repair.

Original authors: Adnan Khan, Abbas Akkasi, Majid Komeili

Published 2026-04-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Adnan Khan, Abbas Akkasi, Majid Komeili

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a friend who is blind how to recognize a dinosaur. You can't just show them a photo; you have to create a raised, bumpy drawing they can feel with their fingers. These are called Tactile Graphics.

But here's the problem: making these drawings is incredibly hard. If a line is too thick, a finger can't tell where the tail ends and the leg begins. If a texture is missing, the skin might feel the same as the ground. Before these drawings reach students, experts have to check them. But until now, the experts could only give a vague grade like "Good" or "Needs Work." They couldn't easily say exactly what was wrong or how to fix it automatically.

This paper introduces TactileEval, a new system that acts like a super-smart, automated quality inspector and editor for these drawings. Here is how it works, broken down into three simple steps:

1. The "Translator" (Turning Expert Knowledge into a Checklist)

Imagine you have a master chef (the expert) who knows exactly why a cake is bad. But you want a line of regular people (crowd workers) to help you check thousands of cakes. You can't ask them to "taste the chemistry." Instead, you give them a simple checklist: "Is the frosting too thick?" "Is the strawberry missing?" "Is the plate dirty?"

The researchers did exactly this. They took the complex, free-text notes from real tactile experts and turned them into a 5-point checklist based on strict rules used by organizations for the blind:

View Angle: Is the animal facing the right way?
Parts: Did we forget the tail or the wings?
Background: Is the picture too cluttered with extra junk?
Texture: Can you feel the difference between fur and feathers?
Line Quality: Are the lines broken or too fuzzy?

They asked thousands of people on Amazon Mechanical Turk to use this checklist on 66 different types of objects (like animals, cars, and food), creating a massive database of 14,000 specific "pass/fail" decisions.

2. The "Detective" (The AI that Learns to Spot Mistakes)

Now that they have this huge checklist database, they trained an AI (a "Vision Transformer" or ViT) to act like a detective.

Think of this AI as a student who has studied the checklist answers. When you show it a new tactile drawing and the original photo, it doesn't just guess "Good" or "Bad." It looks at the drawing and says:

"I am 93% sure the dinosaur's teeth lines are too thick."
"I am 82% sure the tree is missing texture."

The AI got about 86% accuracy in spotting these specific problems. It learned that some things are easy to spot (like a dirty background) and some are hard (like missing tiny parts), just like a human would.

3. The "Editor" (The AI that Fixes the Drawing)

This is the magic part. Usually, if an AI finds a mistake, it just tells you. But TactileEval goes a step further. It acts like a smart editor.

Once the "Detective" AI finds a problem (e.g., "The lines are too thick"), it writes a specific instruction for a powerful image-editing AI (called gpt-image-1).

The Instruction: "Take this dinosaur drawing. The teeth lines are too bold. Please thin them out and make them distinct, but keep the rest of the drawing exactly the same."
The Result: The editing AI re-draws just that specific part, fixing the error without messing up the whole picture.

Why This Matters

Think of the old way of making tactile graphics like hand-crafting a sculpture. You need a master sculptor to check every single piece, and if it's wrong, you have to start over or manually chip away at it.

TactileEval is like giving that sculptor a robot assistant.

The robot scans the sculpture.
It points out, "Hey, this arm is too thick."
It automatically smooths that arm out for you.

This doesn't replace the human experts yet (the paper admits the robot isn't perfect and still needs human review), but it takes the first giant step toward automating the repair process. This means we can eventually produce high-quality, touch-friendly educational materials for blind and visually impaired students much faster and cheaper than ever before.

In short: They built a system that learns exactly what makes a tactile drawing "feel" right, and then uses that knowledge to automatically fix the ones that feel wrong.

1. Problem Statement

Tactile graphics are embossed representations of visual content essential for Blind and Visually Impaired (BVI) learners. Producing high-quality tactile graphics is currently a labor-intensive process requiring expert validation to ensure adherence to standards (e.g., BANA) regarding line quality, texture separation, and viewpoint consistency.

Key Gaps Identified:

Coarse Evaluation: Existing datasets like TactileNet provide only holistic quality ratings (e.g., "Accept," "Minor Edit," "Reject"). These ratings lack actionable signals, failing to specify which perceptual attributes are flawed or how to correct them.
Lack of Automation: There is no automated system capable of diagnosing specific defects or performing targeted repairs on tactile graphics.
Scalability: Reliance on domain experts for every annotation cycle creates a bottleneck for scaling quality assessment across large catalogs of tactile images.

2. Methodology

The authors propose TACTILEEVAL, a three-stage pipeline designed to automate fine-grained evaluation and editing.

A. Taxonomy Construction & Dataset Creation

Expert Mining: The team analyzed free-text comments from the TactileNet dataset to identify recurring tactile failures.
Five-Category Taxonomy: They established a BANA-aligned quality taxonomy with five dimensions:
1. View Match (V): Orientation consistency (Front, Side, Top).
2. Required Parts (P): Completeness of anatomical or functional components.
3. Identity & Background (B): Object identity accuracy and background clutter.
4. Texture Separation (T): Differentiation between adjacent regions.
5. Line Quality (L): Integrity of outlines (broken, fuzzy, or colliding strokes).
Data Collection: Using Amazon Mechanical Turk (AMT), they collected 14,095 structured binary annotations across 66 object classes (organized into 6 families: Animals, Food/Nature, Furniture, Tools, Vehicles, Wearables).
- Protocol: Workers compared natural photos with tactile drawings using family-specific prompts.
- Quality Control: Included gold-sample questions, qualification tests, and a 7-worker voting aggregation with consensus filtering.

B. ViT-Based Quality Evaluator

Architecture: A frozen CLIP ViT-L/14 backbone extracts features from three inputs: the natural image, the tactile image, and the text prompt describing the specific quality option.
Feature Engineering: The model concatenates:
- Natural image embedding.
- Tactile image embedding.
- Element-wise difference (cross-modal discrepancy).
- Text option embedding.
- Result: A 3,072-dimensional feature vector.
Classifier: A lightweight 2-layer MLP (Linear-ReLU-Linear) is trained on these features to perform binary classification (defect present vs. absent) for each of the 30 distinct task families ( $6 \text{ families} \times 5 \text{ dimensions}$ ).
Training Strategy: The ViT backbone remains frozen; only the MLP weights are updated using binary cross-entropy loss.

C. ViT-Guided Automated Editing Pipeline

Diagnosis: The trained probe scores all quality options for a given image pair. The highest-scoring actionable issue is selected as the repair target.
Prompt Engineering: The selected issue code is mapped to a natural-language repair instruction using family-specific templates. These templates include "guardrails" (e.g., maintain clean silhouettes, ensure stroke continuity) to enforce tactile formatting constraints.
Execution: The original tactile image is submitted to gpt-image-1 in "edit mode" (inpainting/editing rather than generation from scratch) using the constructed prompt. This ensures edits are applied to the existing drawing structure.

3. Key Contributions

Fine-Grained Quality Dataset: A new dataset of 14,095 structured annotations covering 30 distinct task families, derived from expert knowledge but scalable via crowdsourcing.
Reproducible ViT Evaluator: A feature probe achieving 85.70% overall test accuracy across 30 tasks, demonstrating that fine-grained tactile quality can be learned from crowd labels without specialist annotators.
Automated Editing Pipeline: A novel framework that translates classifier scores into targeted image edits via LLM-guided prompt engineering, taking a concrete step toward end-to-end tactile repair.
Taxonomy Alignment: A quality framework explicitly aligned with BANA standards, operationalizing abstract accessibility guidelines into concrete, checkable criteria.

4. Results

Evaluation Performance:
- Overall Accuracy: 85.70% on the test set (1,406 records).
- Family Performance: Ranged from 79.51% (Food & Nature) to 89.31% (Vehicles & Flight Systems).
- Task Difficulty: Geometric checks (e.g., View Match, Identity) achieved near-perfect accuracy (1.0). Fine-grained tasks like "Missing Parts" and "Texture Separation" were more challenging (0.50–0.68), reflecting genuine perceptual complexity.
Editing Effectiveness:
- In a study of 15 high-confidence defects, 14/15 edits resulted in a reduction of the ViT issue probability (mean drop of 0.329).
- Case Studies:
  - Line Quality (Dinosaur): Successfully thinned overly thick strokes, though the classifier score change was marginal (highlighting the need for human visual verification).
  - Texture Separation (Tree): Successfully introduced high-contrast fills to separate regions, reducing the issue probability from 0.82 to 0.33.
Zero-Shot Baseline: Unguided generation (text-only or natural-conditioned) failed to maintain species identity or remove background clutter, confirming the necessity of the structured ViT-guided approach.

5. Significance and Future Work

Scalability: By decomposing expert knowledge into plain-language crowd tasks, TACTILEEval removes the bottleneck of requiring tactile accessibility experts for every annotation, enabling large-scale quality assessment.
Actionable AI: Unlike previous holistic rating systems, this pipeline provides specific, actionable repair signals, moving the field toward fully automated tactile graphic correction.
Limitations:
- The editing step relies on the proprietary gpt-image-1 API, limiting full reproducibility.
- The pipeline currently addresses one defect per invocation; multi-defect resolution is future work.
- The system has not yet been validated by BVI participants for actual comprehension outcomes.
Future Directions: Calibration of confidence thresholds to address probability inflation, adding lightweight decoders for self-contained rationales, and longitudinal studies with BVI users to validate comprehension improvements.

Conclusion: TACTILEEval represents a significant step forward in accessibility technology, bridging the gap between automated generation and human-quality tactile graphics through a structured, data-driven, and editable pipeline. All code, data, and models are open-sourced.