Leveraging GenAI for Segmenting and Labeling Centuries-old Technical Documents

This paper presents a novel approach to automating the curation and retrieval of 16th and 17th-century shipbuilding treatises by leveraging generative AI models like SAM2, Florence2, and ChatGPT, enhanced with specialized nautical ontologies, to overcome the challenges of segmenting and labeling scarce, domain-specific historical documents.

The Gist

Imagine you have a massive, dusty library filled with ancient, handwritten blueprints for ships. These books are hundreds of years old, written in different languages, and the drawings are intricate, faded, and full of technical terms that only a master shipwright from the 1600s would understand.

Now, imagine you want to build a modern, searchable digital catalog for this library. You want a computer to look at a picture of a ship, point to every single part (the mast, the rudder, the specific type of knot), and tell you exactly what it is.

The Problem:
Modern computers are great at recognizing things in today's photos. If you show a computer a picture of a cat on a car, it knows exactly what a "cat" and a "car" are because it has seen millions of them on the internet. But if you show it a 400-year-old drawing of a ship, the computer gets confused. It doesn't know what a "rider frame" is, or how to tell the difference between a "keel" and a "keelson" in a sketch that looks like a messy doodle to a modern eye. It's like trying to teach a toddler to read a complex engineering manual.

The Solution: A Team of AI Specialists
The authors of this paper tried to solve this by creating a "dream team" of Artificial Intelligence tools to act as a digital curator. Think of their process like a three-step assembly line in a factory:

1. The "Cutting" Machine (Segmentation):
   First, they needed to separate the different parts of the ship drawing. Imagine a drawing of a ship where the hull, the sails, and the ropes are all drawn on top of each other. The computer uses a tool called SAM2 (think of it as a super-precise digital pair of scissors) to cut the image into pieces. It draws outlines around every single part, isolating the "wheel" from the "axle," even if they are touching.

2. The "Labeling" Machine (Recognition):
   Once the pieces are cut out, the computer needs to know what they are. This is where they tried different "brain" tools.

   • The Old Way: They tried using standard AI that just guesses based on general knowledge. It was like asking a tourist to identify parts of a spaceship; they might guess "wheel" when it's actually a "thruster."
   • The New Way: They used GenAI (like ChatGPT) and specialized tools. But here's the catch: if you just ask the AI, "What is this?" it might give you a wrong answer because it doesn't know the context. It might think a "rider frame" is part of a motorcycle!

3. The "Expert" Dictionary (The Ontology & Glossary):
   This is the secret sauce. The researchers didn't just let the AI guess. They gave the AI a specialized dictionary and a rulebook (called an ontology) written by real human experts on shipbuilding.

   • Think of this as giving the AI a cheat sheet. Instead of saying, "What is this?" they tell the AI, "Look at this shape. In our rulebook, this shape is called a 'Rider Frame,' and it is always found at the back of the ship."
   • This forces the AI to stop guessing and start using the correct, historical vocabulary.

The Results: A Work in Progress
The experiment showed that this "Team Approach" works much better than just letting the AI guess on its own.

• The Good: The computer can now identify more parts of the ship and give them the correct, fancy names. It's like upgrading from a toddler's vocabulary to a professor's vocabulary.
• The Bad: It's not perfect yet. Sometimes the computer still gets confused. It might label a pulley as an "axis" or think a book edge is a "sharpened object." It's like a very smart student who knows the definitions but sometimes applies them to the wrong picture.

Why Does This Matter?
Why spend so much time teaching a computer to read old ship drawings?

• Preservation: These books are priceless. If they are lost or damaged, that knowledge disappears forever.
• Accessibility: Right now, only a few experts can read these books. If we can teach a computer to understand them, we can build a search engine where a student, a historian, or a curious grandparent can type "show me all the ships with red sails" and instantly find the right pages.
• Reconstruction: By understanding exactly how these ships were built from the drawings, we can help archaeologists rebuild sunken ships or understand how our ancestors explored the world.

In a Nutshell:
The paper is about teaching a computer to become a digital shipwright. By combining a "digital scissors" to cut up the images, a "smart brain" to guess what they are, and a "human expert's dictionary" to correct the guesses, they are making it possible to unlock the secrets of centuries-old shipbuilding manuals for everyone. It's a bit like giving a time machine a library card.

Technical Summary

1. Problem Statement

The paper addresses the challenge of automating the curation, cataloging, and retrieval of digitized historical technical documents, specifically shipbuilding treatises from the 16th and 17th centuries (Age of Exploration). While modern image segmentation and recognition (e.g., Meta's SAM2) perform exceptionally well on contemporary photographs due to vast training datasets, they face significant hurdles when applied to specialized historical domains:

• Data Scarcity: There is a lack of sufficient training data for centuries-old, domain-specific illustrations.
• Specialized Knowledge: These documents contain complex technical illustrations (e.g., nautical architecture) that require deep domain expertise to interpret. Standard AI models often fail to distinguish between composite objects (e.g., identifying specific ship parts like a "keel" vs. the whole "ship") or misinterpret technical diagrams due to a lack of contextual understanding.
• Goal: The authors aim to improve the indexing and semantic retrieval of these priceless collections to make them accessible to scholars and the public.

2. Methodology

The authors propose a three-step pipeline that integrates computer vision (CV) models with Generative AI (GenAI) and domain-specific knowledge bases (Ontologies and Glossaries).

A. Image Segmentation

• Primary Tool: SAM2 (Segment Anything Model 2) is used to identify and isolate specific objects and regions within the historical illustrations.
• Comparison: SAM2 is compared against the traditional Watershed algorithm (via OpenCV). Results indicate SAM2 provides superior granularity, detecting fine details (e.g., text segments, bleed-through, specific geometric shapes) that Watershed misses.
• Granularity: The system aims for high-granularity segmentation (identifying individual ship parts like "Alifez" or "Quilha") rather than just the composite object (the entire ship).

B. Labeling and Object Recognition

The paper evaluates four distinct methodological approaches for generating labels and bounding boxes:

1. BLIP + NLTK + GroundingDINO: Uses BLIP for captions, NLTK for tokenization/lemmatization, and GroundingDINO for bounding boxes.
2. Tag2Text: Replaced BLIP/NLTK but failed due to reliance on a pre-included plaintext list of terms, limiting flexibility.
3. RAM (Recognize Anything Model) + GroundingDINO: An improved version of Tag2Text with a better vocabulary.
4. GenAI-Enhanced Pipeline (Proposed):
   • SAM2 segments the image.
   • ChatGPT is used to generate detailed definitions for terms found in a specialized glossary.
   • RAM utilizes these ChatGPT-generated definitions to create embeddings for open-set inference, allowing it to recognize terms not strictly in its pre-trained vocabulary.
   • GroundingDINO assigns bounding boxes to the recognized labels.

C. Knowledge Integration (Ontology & Glossary)

To mitigate the "hallucination" and domain ignorance of GenAI, the authors integrate:

• ontoShip: A specialized ontology containing ~250 concepts of wooden ship architecture (e.g., hull components, fastening systems) with spatial and functional properties.
• glosShip: A curated multilingual glossary of nautical terms.
• Strategy: Instead of using generic prompts, the system leverages definitions from glosShip and concepts from ontoShip to craft precise prompts for ChatGPT. This ensures the AI understands the context (e.g., distinguishing a "rider frame" in a ship context vs. a motorcycle context).

3. Key Contributions

• Domain-Specific Pipeline: A novel workflow combining state-of-the-art foundation models (SAM2, Florence-2, RAM) with domain-specific knowledge graphs (ontoShip) to handle low-resource, high-specialization historical data.
• GenAI Prompt Engineering: Demonstrating that carefully crafted prompts using specialized glossary definitions significantly improve the accuracy of GenAI in technical domains compared to generic prompting.
• Comparative Analysis: A systematic evaluation of different segmentation and labeling combinations (SAM2 vs. Watershed; BLIP vs. RAM vs. ChatGPT-enhanced RAM).
• Ontology-Driven Grounding: Proposing the use of ontologies to act as a "knowledge filter" for GenAI, reducing false positives in technical labeling.

4. Results and Discussion

• Segmentation: SAM2 significantly outperformed the Watershed algorithm, successfully detecting fine details (text, bleed-through, specific geometric shapes) in 17th-century Portuguese treatises.
• Labeling Accuracy:
  • ChatGPT-enhanced methods showed the most promise in identifying a higher number of objects and providing detailed descriptions.
  • Challenges: Without domain context, GenAI produced false positives (e.g., labeling pulleys as "axes" or book edges as "sharpened objects").
  • Contextual Correction: Providing specific nautical context to ChatGPT (e.g., specifying "in a shipbuilding context") corrected definitions and improved relevance.
• Granularity: The pipeline successfully moved from labeling the whole ship to labeling specific components (e.g., Codaste, Escarva), which is crucial for detailed retrieval.

5. Significance and Future Work

• Cultural Heritage Preservation: The approach offers a scalable solution for automating the curation of large, unstructured historical archives (the authors plan to process 14 treatises with ~4,000 images in multiple languages).
• Information Retrieval: By moving from generic tags to ontology-aligned labels, the system enables precise semantic search, allowing researchers to query specific ship components across centuries of documentation.
• Future Directions:
  • Benchmarking: Conducting a comprehensive comparison between Microsoft Florence-2 and other models (RAM, BLIP) using the specialized glossary.
  • Metric Development: Designing metrics to quantify the difference between AI-generated labels and human-curated "ground truth" (glosShip/ontoShip).
  • Pipeline Optimization: Testing hybrid approaches, such as using ChatGPT to generate captions that are then fed into Florence-2 for grounding.

In conclusion, the paper demonstrates that while off-the-shelf AI models struggle with specialized historical documents, marrying GenAI with curated ontologies and glossaries creates a robust framework for automating the analysis of technical heritage, bridging the gap between modern computer vision and historical domain expertise.