Deep Learning-Based Tongue Pathology Detection: Mapping Fine-Grained Computer Vision Features to Ayurvedic Diagnostic Heuristics

Technical Summary: Deep Learning-Based Tongue Pathology Detection

Problem Statement
Traditional Ayurvedic medicine relies on tongue examination to classify individuals into constitutional types (doshas: Vata, Pitta, Kapha) and provide personalized wellness recommendations. However, this assessment is currently dependent on practitioner expertise, visual experience, and subjective interpretation, leading to variability and limited reproducibility. While recent advances in artificial intelligence (AI) and computer vision have transformed medical image analysis, existing studies in tongue diagnosis primarily focus on disease classification within Traditional Chinese Medicine (TCM) or rely heavily on questionnaire-based assessments. There is a notable scarcity of lightweight, mobile-friendly systems that integrate fine-grained tongue feature detection with Ayurvedic constitutional assessment using on-device inference. Furthermore, challenges such as limited public benchmark datasets, inconsistent annotation protocols, and the lack of standardized evaluation methodologies hinder the development of robust, interpretable systems for this domain.

Methodology
The paper proposes "Jihwa," a lightweight edge-AI mobile framework designed for non-invasive tongue feature analysis and rule-based Ayurvedic wellness assessment. The system architecture consists of three primary layers:

1. Perception Layer (Computer Vision):

   • Model: The system employs a YOLO11n object detection model, chosen for its balance of accuracy and computational efficiency suitable for mobile deployment.
   • Training: The model was trained via transfer learning on the TMC-Tongue dataset, which contains 6,719 annotated images across 20 tongue-feature categories (including tongue body color, coating, fissures, teeth marks, swelling, and organ-zone indicators).
   • Preprocessing: Images were resized to 640×640 pixels. Data augmentation techniques (scaling, translation, horizontal flipping, and controlled HSV transformations) were applied to enhance robustness while preserving clinically meaningful color patterns.
   • Inference: The trained model is converted to TensorFlow Lite format, enabling real-time, on-device inference without transmitting images to external servers. The model outputs bounding boxes, class labels, and confidence scores, which are refined using confidence thresholds and Non-Maximum Suppression (NMS).

2. Interpretation Layer (Multi-Modal Fusion):

   • Questionnaire: Users complete a 22-question self-assessment covering physiological and behavioral traits (digestion, sleep, energy, etc.), which are mapped to Vata, Pitta, and Kapha scores.
   • Fusion Mechanism: The framework utilizes a transparent, rule-based weighted fusion strategy. Visual features extracted by the YOLO11n model contribute 60% to the final assessment, while questionnaire responses contribute 40%.
   • Scoring: Detected tongue features are assigned Ayurvedic weights based on predefined diagnostic heuristics. The visual score ($S_{visual}$) is calculated by summing the product of feature weights and confidence scores. This is combined with the normalized questionnaire score ($S_{questionnaire}$) using the formula: $S_{final} = 0.6S_{visual} + 0.4S_{questionnaire}$.
   • Classification: The fused scores are normalized to generate percentage-based values for the three doshas. The system identifies a dominant constitution or a mixed constitution (e.g., Vata-Pitta) based on score thresholds, supporting seven distinct constitution types.

3. Recommendation Layer:

   • Based on the identified constitution, the system generates personalized wellness recommendations, including dietary guidance, hydration strategies, and an eight-week staged lifestyle plan. All recommendations are generated on-device to preserve user privacy.

Key Contributions

• Integration of Modalities: The paper presents a unified framework that combines fine-grained object detection of tongue features with structured questionnaire-based reasoning, a combination rarely found in existing edge-AI health applications.
• Edge-AI Implementation: Unlike cloud-based approaches, the system executes the YOLO11n model entirely on the user's mobile device, ensuring low latency, data privacy, and independence from internet connectivity.
• Interpretability: The framework distinguishes between the machine learning component (feature detection) and the rule-based wellness interpretation. This transparency allows for clear mapping of visual features to Ayurvedic heuristics, avoiding the "black box" nature of end-to-end deep learning diagnosis.
• Dataset Utilization: The work leverages the TMC-Tongue dataset for training a detector capable of identifying 20 specific tongue characteristics, addressing the gap in fine-grained feature detection for Ayurvedic contexts.

Experimental Results
The YOLO11n model was trained for 150 epochs on the TMC-Tongue dataset. Performance was evaluated using standard object detection metrics on a held-out test set:

• Precision: 0.424
• Recall: 0.318
• mAP@50: 0.293
• mAP@50-95: 0.232

The results indicate that the model successfully learned discriminative tongue characteristics. Analysis of the confusion matrix revealed that most classes exhibited strong diagonal dominance, with misclassifications occurring primarily among visually similar features (e.g., specific coating or fissure patterns). The training curves demonstrated stable convergence, with losses decreasing and metrics improving over epochs. The F1-confidence curve identified an optimal threshold region where precision and recall were balanced.

Significance and Claims
The author positions this work not as a replacement for clinical expertise, but as a tool for accessible, personalized wellness monitoring. The significance of the paper lies in its demonstration of a transparent, computationally efficient framework that bridges traditional Ayurvedic diagnostic heuristics with modern deep learning. By clearly separating the validated machine learning detection from the rule-based interpretation, the study provides a foundation for future research at the intersection of edge AI, digital health, and computational traditional medicine. The author claims that this approach offers a practical solution for overcoming the subjectivity of traditional tongue diagnosis while maintaining the privacy and accessibility required for everyday health applications.