Image-based Joint-level Detection for Inflammation in Rheumatoid Arthritis from Small and Imbalanced Data

1. Problem Statement

The paper addresses the critical need for remote, accessible diagnosis of Rheumatoid Arthritis (RA) joint inflammation. While early detection is vital to prevent irreversible joint damage, current standard methods (MRI, ultrasound, specialist physical exams) are resource-intensive and often inaccessible to patients in remote areas or those without immediate specialist access.

The authors propose using RGB images (captured via standard cameras/phones) to detect active synovial inflammation at the joint level. However, this task faces three significant technical challenges:

1. Data Scarcity: It is difficult to collect large-scale datasets of affected hands; positive samples are extremely rare.
2. Severe Class Imbalance: Even within RA patient datasets, inflammation is localized. Most joints are healthy, creating a massive imbalance between negative (healthy) and positive (inflamed) samples at the joint level.
3. Label Reliability: Visual inspection by rheumatologists often disagrees with the actual presence of inflammation. Existing datasets often rely on subjective visual labels rather than objective ground truth (ultrasound), leading to noisy training data.

2. Methodology

The authors propose a Global-Local Encoder Framework combined with a Two-Stage Training Strategy to overcome data limitations.

A. Network Architecture

The model utilizes a dual-encoder approach based on ResNet-18:

• Global Encoder: Processes the full hand image ($X$) to capture overall structural deformities and global context.
• Local Encoder: Processes cropped patch images ($X_{local}$) of individual finger joints. Joint positions are detected using a landmark estimation framework (MediaPipe) to extract specific regions.
• Integration: Features from both encoders are concatenated and fed into an MLP (Multi-Layer Perceptron) header to output binary inflammation labels for each joint.

B. Two-Stage Training Strategy

To address the lack of positive samples and class imbalance:

1. Self-Supervised Pre-training: Both encoders are pre-trained on a large-scale dataset of 11,000 healthy hand images using DINO (a self-supervised learning method). This allows the model to learn robust anatomical features without requiring disease labels.
2. Fine-tuning with Focal Loss: The pre-trained encoders are frozen. Only the classifier head is fine-tuned on the specific RA dataset. To handle the extreme class imbalance, the authors replace standard Binary Cross-Entropy (BCE) loss with Focal Loss. This re-weights the loss function to focus training on hard-to-classify (minority) positive samples, preventing the model from simply predicting "healthy" for all inputs.

3. Key Contributions

1. High-Reliability Dataset Construction: The authors constructed a new RGB hand image dataset where inflammation labels are derived from ultrasound findings (Gray Scale > 1 and Power Doppler > 0) rather than subjective visual inspection. This bridges the gap between external appearance and internal pathology.
2. Novel Framework: A scalable deep learning framework integrating global and local features specifically designed for small, imbalanced medical datasets.
3. Imbalance-Aware Optimization: The successful application of self-supervised pre-training on healthy data combined with Focal Loss to achieve high sensitivity despite the scarcity of positive samples.
4. Benchmarking: A comprehensive evaluation showing that visual diagnosis of RA inflammation is extremely difficult even for human experts when relying solely on images.

4. Experimental Results

The study was conducted on a dataset of 222 hand images from 68 patients, with annotations based on ultrasound.

• Performance vs. Baselines: The proposed method significantly outperformed baseline models (ResNet-18, ResNet-50, Phatak et al.'s Inception-ResNetv2, and Vision Transformers).
  • F1-Score: Improved by 0.20 points over the baseline.
  • Gmean: Improved by 0.25 points over the baseline.
  • Recall: Achieved 0.478, significantly higher than baselines (which ranged from 0.129 to 0.170).
• Ablation Studies:
  • Removing DINO pre-training dropped the F1-score from 0.420 to 0.352.
  • Removing Focal Loss caused a catastrophic drop in Recall (0.115) and F1-score (0.158), confirming its necessity for handling imbalance.
  • Removing the Global/Local architecture reduced Gmean from 0.605 to 0.517.
• Comparison with Human Experts:
  • Three rheumatologists evaluated the same joint images. Their average F1-score was only 0.305 and Precision was 0.214.
  • While experts had high Recall (~0.58), their low precision indicates that visual inspection alone is a poor predictor of ultrasound-confirmed inflammation. The proposed model (F1=0.420) outperformed the human average.
• Model Behavior: Large models like ViT failed to converge (predicting all negatives) due to overfitting on the small dataset, highlighting the necessity of the proposed lightweight, pre-trained approach.

5. Significance and Conclusion

This paper demonstrates that RGB-based remote monitoring for RA is feasible but requires specialized handling of data scarcity and imbalance.

• Clinical Impact: The proposed system offers a tool for patients to self-assess joint inflammation at home, potentially reducing the burden on specialists and enabling earlier intervention.
• Methodological Insight: It proves that self-supervised pre-training on healthy data, combined with imbalance-aware loss functions, is superior to standard supervised learning for small medical datasets.
• Limitations & Future Work: The current method assumes inflammation is visible in RGB images. The authors acknowledge that in extremely early stages, changes may be invisible to the naked eye, which remains a challenge for future research.

In summary, this work provides a robust, objective, and scalable solution for detecting RA inflammation using accessible technology, outperforming both standard deep learning baselines and human visual assessment in the context of image-only diagnosis.