LLM-Driven AutoML for Cross-Lingual Handwritten OCR: Closed-Loop Neural Architecture Search with GPT-5, GPT-4o, and Claude Sonnet 4

Technical Summary: LLM-Driven AutoML for Cross-Lingual Handwritten OCR

Problem Statement
Handwritten text recognition (HTR) across diverse scripts, such as Arabic, English, and Persian, remains a significant challenge in machine learning due to unique visual complexities, intricate stroke patterns, and high variability in writing styles. Traditional approaches rely heavily on expert-guided model design, extensive manual preprocessing, and iterative hyperparameter tuning. These methods are often time-consuming, difficult to scale to new scripts, and prone to inconsistent outcomes. While recent advancements in Neural Architecture Search (NAS) and Large Language Models (LLMs) have shown promise, their application as independent, closed-loop agents for generating and refining deep learning architectures specifically for cross-lingual handwritten OCR has remained largely unexplored.

Methodology
The authors propose a fully automated, end-to-end pipeline that utilizes Large Language Models (specifically GPT-5, GPT-4o, and Claude Sonnet 4) as autonomous "AI architects." The system operates through a closed-loop iterative process comprising four core stages:

1. Data Collection and Augmentation: The pipeline utilizes the EMNIST (English), SADRI (Persian), and AHCD (Arabic) datasets. Data is standardized into tensor formats with stratified sampling. A lightweight augmentation strategy (random rotations, shifts, zooms) is applied during training to enhance generalization without significant computational overhead.
2. LLM-Driven Architecture Proposal: At the start of each trial, the system prompts the LLMs with dataset metadata (class count, image dimensions) and, in subsequent iterations, performance metrics from previous trials. The LLMs respond with a structured JSON specification detailing the neural network architecture (e.g., Conv2D, BatchNorm, Dropout, Transformer blocks) and training hyperparameters (optimizer, learning rate, batch size).
3. Automated Model Construction and Training: The JSON specifications are parsed and automatically converted into executable Keras models. These models range from standard CNNs to hybrid architectures incorporating Vision Transformer components. Models are compiled with categorical cross-entropy loss and trained without human intervention.
4. Evaluation and Feedback Loop: Upon completion of training, the system evaluates the model on test, validation, and training sets, recording accuracy, parameter count, and inference latency. These metrics are fed back into the LLM as a structured summary. The LLM uses this feedback to refine its architectural proposals in the next iteration, creating a self-improving loop that converges toward optimal designs for each specific script.

Key Contributions

• Unified Cross-Script Framework: The work introduces a single framework capable of recognizing Arabic, English, and Persian scripts, adapting to the specific structural and visual complexities of each without manual reconfiguration.
• LLM as Generative Agent: Unlike prior works that use LLMs merely for recognition or as static tools, this approach employs GPT-5, GPT-4o, and Claude Sonnet 4 as standalone agents responsible for the entire lifecycle of model discovery, from proposal to refinement.
• Closed-Loop Iterative Refinement: The system implements a dynamic feedback loop where LLMs learn from trial-by-trial results to adjust layer types, depth, width, and hyperparameters, eliminating the need for manual tuning.
• Transparency and Reproducibility: The pipeline rigorously documents every trial, saving architecture configurations and performance metrics in structured formats (JSON, CSV) to ensure full reproducibility.

Experimental Results
The pipeline was evaluated over thirty independent trials for each of the three scripts and three LLMs. Key findings include:

• Accuracy: The system consistently discovered models with high test accuracy. GPT-4o achieved the highest mean accuracy for Arabic (0.959), while Claude Sonnet 4 led for Persian (0.946). GPT-5 achieved a best-case trial accuracy of 0.981 on Arabic. Average accuracies across all models and scripts generally exceeded 93%.
• Efficiency and Latency: GPT-5 generated the most compact architectures (0.88M to 1.35M parameters), while Claude Sonnet 4 produced larger models (up to 9.28M parameters). Despite significant variations in parameter counts, inference latency remained stable and real-time suitable (approximately 40–44 ms), suggesting that runtime cost is driven more by architectural depth than raw parameter size.
• Generalization: Validation curves closely tracked training curves (differences typically within 1–2%), indicating strong generalization and effective regularization without overfitting.
• Comparison: Compared to prior work (e.g., Cerescu & Bumbu, 2024), which treated LLMs as direct recognizers for low-resource scripts, this framework utilizes LLMs as generative agents for automated model design, achieving higher accuracy and full automation across multiple languages.

Significance and Claims
The paper claims that this work demonstrates the capability of Large Language Models to transcend their traditional role in language processing and function as independent designers for machine learning systems. By automating the discovery of neural architectures for cross-lingual handwritten OCR, the framework offers a scalable, script-agnostic, and fully automatic solution. The authors assert that this approach significantly simplifies OCR model development, removes the dependency on domain-specific heuristics and expert intervention, and opens the door for rapid, adaptable deployment of OCR technology across diverse languages and domains. The results validate that LLM-driven architecture search can effectively balance predictive performance, inference efficiency, and model complexity.