🤖 AI

A Reliability Evaluation of Hybrid Deterministic-LLM Based Approaches for Academic Course Registration PDF Information Extraction

This study demonstrates that a hybrid approach combining the Camelot PDF extraction pipeline with a Qwen 2.5:14b LLM fallback achieves superior accuracy and computational efficiency for extracting academic course registration data from PDFs on consumer-grade hardware compared to using LLMs alone.

Original authors: Muhammad Anis Al Hilmi, Neelansh Khare, Noel Framil Iglesias

Published 2026-04-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Muhammad Anis Al Hilmi, Neelansh Khare, Noel Framil Iglesias

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a university administrator drowning in a sea of paper (or rather, PDFs). Every semester, thousands of students submit their "Study Plans" (called KRS in Indonesia), which are essentially lists of the classes they are taking and the teachers assigned to them.

The university needs to read these lists to calculate grades, track attendance, and plan budgets. But here's the problem: the university's computer system is old, the IT team is small, and they can't just ask the main database for the data because of privacy rules and red tape. They need a way to "read" these PDFs and turn them into digital data automatically.

This paper is like a recipe book for building a robot reader that can do this job on a regular, non-expensive laptop (no super-computers allowed).

Here is the breakdown of their experiment, explained with everyday analogies:

1. The Three "Readers" They Tested

The researchers tried three different ways to get the robot to read the PDFs:

The "Pure AI" Reader (LLM-Only):
- The Analogy: Imagine hiring a brilliant, super-smart intern who has read the entire internet. You hand them a messy PDF and say, "Tell me what's in here."
- The Result: The intern is very smart and understands context, but they are slow. They take about 1.5 to 2 minutes to read just one page. If you have 800 pages, you'd be waiting days. Also, sometimes the intern gets confused by weird formatting or typos.
The "Hybrid" Reader (Regex + AI):
- The Analogy: This is like hiring the smart intern again, but this time, you give them a checklist (a set of strict rules) for the easy stuff.
- How it works: A "regex" (a computer rule) acts like a scanner that instantly grabs the easy, fixed information (like the student's name or ID number) because it always appears in the same spot. The AI only has to read the messy middle part (the list of courses).
- The Result: This was faster and more reliable for the easy stuff, but the AI still took too long to read the course lists.
The "Smart Filing Cabinet" Reader (Camelot + AI Fallback):
- The Analogy: This is the winner. Imagine a specialized filing cabinet that knows exactly how to pull out tables from a document. It's like a specialized librarian who can instantly spot a grid of data and pull it out.
- The Strategy: The system tries the "Smart Librarian" (Camelot) first. If the Librarian succeeds, it's done in less than 1 second. If the Librarian gets stuck (maybe the table lines are faint), then it calls the "Smart Intern" (the AI) to help finish the job.
- The Result: This was the champion. It was incredibly fast (under 1 second per PDF) and almost perfectly accurate.

2. The Models (The "Brains")

They tested three different AI models (Gemma, Phi, and Qwen) running on a standard laptop.

The Metaphor: Think of these as different types of students.
- Gemma: A hardworking student who needs a lot of help (rules) to get things right. Without the checklist, they fail.
- Phi: A student who is good at some things but gets confused when names are repeated or when the layout changes.
- Qwen: The star student. They were the most consistent, making the fewest mistakes across all tests, even when the PDFs were tricky.

3. The Big Challenges

The researchers faced a few specific hurdles, like trying to read a document where:

The "Ligature" Trap: Sometimes the font makes two letters look like one (like "fi" looking like a single symbol). The robot had to learn to unscramble these.
The "Advisor" Confusion: The PDF lists a student's advisor at the top, but the robot was told not to count them as a teacher for the courses. One AI model (Phi) kept accidentally deleting real teachers because it thought they were the advisor!
The "Long Name" Problem: Some student names were so long they wrapped around to the next line. The AI sometimes cut the name in half.

4. The Final Verdict

The paper concludes that you shouldn't rely on just one method.

Don't just use the AI: It's too slow for big jobs.
Don't just use rules: They are too rigid and break when the document changes slightly.
Do use a "Progressive Fallback" system:
1. Try the fast, specialized tool (Camelot) first.
2. If that fails, try the AI as a backup.
3. Use strict rules for the simple stuff (names/IDs).

The Takeaway:
By combining a fast, specialized tool with a smart AI backup, the researchers built a system that can process hundreds of student documents in minutes on a regular laptop, keeping the data private and secure. It's the difference between hiring one slow genius versus hiring a team with a fast scanner and a smart assistant working together.

1. Problem Statement

The study addresses the challenge of extracting structured data from Indonesian academic "Study Plan" (KRS) PDF documents. While KRS data is crucial for administrative and analytical purposes (e.g., attendance, grading, teaching evaluation), accessing it directly from campus databases is often restricted due to legacy code limitations, permission barriers, and data privacy concerns.

The Challenge: KRS PDFs are text-based (not scanned images) but exhibit significant structural variations across different study programs (e.g., varying table layouts, Unicode ligatures like "fi" vs. "f", and inconsistent metadata).
The Constraint: The extraction system must operate locally on consumer-grade hardware (CPU-only, no GPU) to ensure data privacy and avoid external API costs, while maintaining high accuracy and throughput.

2. Methodology

The researchers evaluated three distinct extraction strategies using a dataset of 140 PDFs (for initial model comparison) and 860 PDFs (for pipeline validation).

A. Experimental Setup

Hardware: Advan Workplus laptop (Ryzen 5 6000H, 16GB RAM, Windows 11) running without a GPU.
Models: Three 12–14B parameter models run locally via Ollama:
- Gemma 3:12b
- Phi-4:14b
- Qwen 2.5:14b
Target Data: Metadata (student info), Course Lists, and Lecturer Lists.
Evaluation Metrics:
- Exact Match (EM): Strict string comparison.
- Levenshtein Similarity (LS): Tolerant of minor errors (spaces, punctuation) with a threshold of 0.7.

B. Proposed Approaches

LLM-Only: Direct prompt-based extraction using few-shot prompting.
Hybrid Deterministic–LLM:
- Uses Regex for stable metadata extraction (fast, deterministic).
- Uses LLM for complex table parsing (courses/lecturers).
Camelot-Based Pipeline with LLM Fallback:
- Regex for metadata.
- Camelot Library (layout-aware) for table extraction using two "flavors":
  - Lattice: For tables with clear borders.
  - Stream: For tables without borders.
- LLM Fallback: If Camelot fails (e.g., multi-line text issues), the LLM is triggered as a last resort.

3. Key Contributions

Context-Specific Evaluation: Fills a research gap by evaluating KRS extraction specifically within the Indonesian academic context, addressing unique challenges like Unicode ligatures and varying study program structures.
Privacy-Preserving Architecture: Demonstrates a viable pipeline for processing sensitive student data entirely on consumer-grade CPUs, eliminating the need for cloud-based APIs.
Hybrid Strategy Validation: Proves that combining deterministic rules (Regex/Camelot) with probabilistic models (LLM) significantly outperforms LLM-only approaches in both accuracy and efficiency.
Model Benchmarking: Provides a comparative analysis of Gemma 3, Phi 4, and Qwen 2.5 on CPU-only hardware, identifying Qwen 2.5 as the most robust for this specific task.

4. Results

A. Model Performance (140 PDFs)

Qwen 2.5:14b demonstrated the most consistent performance across all modes and study programs, achieving near-perfect scores (EM/LS $\approx$ 1.00).
Gemma 3:12b failed significantly in "LLM-only" mode (scoring ~0.68–0.82) but achieved perfect scores (1.00) in Hybrid mode, proving it relies heavily on deterministic pre-processing.
Phi 4:14b showed mixed results; it struggled with lecturer lists in "LLM-only" mode and often incorrectly removed duplicate lecturer names (violating the prompt rule that course count must equal lecturer count).

B. Throughput and Efficiency

LLM-Only/Hybrid: Required 1.5 to 1.8 minutes per PDF (approx. 0.54–0.69 PDFs/min). This is computationally expensive for large-scale processing.
Camelot Pipeline (860 PDFs):
- Achieved <1 second per PDF for the vast majority of documents.
- Accuracy: Achieved EM/LS scores of 0.99–1.00.
- Fallback Efficiency: The LLM fallback was only triggered for ~9 out of 860 documents (when Camelot failed on complex layouts), reducing the average LLM inference time to ~30 seconds only for those specific cases.

C. Specific Failure Modes

Unicode/Ligatures: Caused minor mismatches in names (e.g., "fi" vs "f").
Multi-line Text: Camelot's "Lattice" mode sometimes failed to capture long course titles that wrapped to a second line, requiring the LLM fallback.
Phi 4 Specifics: Tended to delete duplicate lecturer names, breaking the 1:1 mapping between courses and lecturers.

5. Significance and Conclusion

The study concludes that a progressive fallback strategy (Regex $\to$ Camelot Lattice/Stream $\to$ LLM) offers the optimal balance for academic document extraction:

Reliability: Deterministic methods handle stable metadata and standard tables with near-perfect accuracy.
Efficiency: The pipeline processes documents in under 1 second for 99% of cases, making it scalable for large datasets.
Robustness: The LLM acts as a safety net for edge cases (complex layouts, multi-line text) without incurring the high computational cost of running it on every document.

Future Work suggested includes:

GPU acceleration to further reduce inference time.
Domain-specific fine-tuning of models for Indonesian academic terminology.
Expanding the pipeline to handle scanned/OCR-based PDFs.

This research validates that hybrid deterministic-neural approaches are a superior, privacy-compliant, and computationally efficient solution for extracting data from structured academic documents in resource-constrained environments.