💻 computer science

Foreseeing the Invisible: Amodal Reconstruction of Leaf Fossil Images

This paper introduces AmodalDINO, a multi-head dense-prediction model that reconstructs complete leaf fossils from partial images without requiring prior visible masks, achieving high accuracy through fine-tuned DINOv3 and auxiliary venation heads while enabling practical offline deployment and surface area estimation.

Original authors: Liuxiang Yue, Ailin Zhang, Ziyue Zhao, Yikun Duan

Published 2026-08-06

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Liuxiang Yue, Ailin Zhang, Ziyue Zhao, Yikun Duan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a detective trying to solve a crime, but the only clue they have is a torn-up photograph where half the suspect's face is covered by a rock. In the world of paleobotany—the study of ancient plants—scientists face this exact puzzle every day. They study fossilized leaves, but nature is messy. Over millions of years, sedimentary rock buries parts of the leaf, cracks split them apart, and weathering eats away the edges. What remains is rarely the whole picture; it's just a fragment. Yet, to understand the past, scientists need the entire leaf: its full shape, its outline, and the intricate network of veins that once carried water. This is where a concept called "amodal reconstruction" comes in. Think of it as the brain's ability to "see" the whole object even when part of it is hidden behind a couch or a wall. For decades, computers have struggled to do this for complex, organic shapes like leaves, especially when they are buried in stone and the computer doesn't even know where the visible part starts.

This paper introduces a clever new tool called AmodalDINO that acts like a super-powered imagination engine for fossil leaves. Instead of just tracing the parts of the leaf that are visible, the model predicts the invisible parts hidden under the rock, effectively "hallucinating" the missing tissue in a way that is scientifically accurate. The researchers trained this AI using thousands of computer-generated images of broken leaves, teaching it to recognize the structural patterns of plant veins and leaf shapes. The result is a system that can take a single photo of a fossil, guess which parts are covered by rock, and draw the complete leaf, including its main stem and tiny side veins, without needing a human to point out where the leaf begins.

The Magic of "Foreseeing the Invisible"

Meet AmodalDINO, a digital detective designed to solve the mystery of the missing leaf. Imagine you find a leaf in a museum case, but it's stuck inside a chunk of gray rock. You can see the tip and maybe a bit of the side, but the rest is buried. A normal computer vision program would just draw a box around the visible green part and say, "That's the leaf." But a paleobotanist needs to know the whole shape to identify the species or guess what the climate was like millions of years ago. They need to know what the leaf would look like if the rock weren't there.

This is where AmodalDINO steps in. It doesn't just look at what it can see; it predicts what it can't see. It's like looking at a puzzle with half the pieces missing and being able to draw in the missing pieces so perfectly that you could almost believe they were always there. The model outputs four different "masks" (digital outlines) at once:

The Visible Leaf: The part you can actually see.
The Complete Leaf: The full shape, including the parts buried in the rock.
The Main Vein: The central "spine" of the leaf.
The Fine Veins: The tiny, hair-like branches that spread out.

How It Works: The "Unfreezing" Trick

The secret sauce of this paper isn't just that the AI is smart; it's how the researchers taught it to be smart. They started with a very powerful pre-trained AI called DINOv3, which is like a student who has already read every book in the library but hasn't learned how to solve this specific puzzle yet.

Usually, when scientists use a pre-trained AI, they "freeze" its brain. They keep the student's knowledge locked in place and only train a small add-on to solve the new problem. The authors of this paper tried that, and it failed. The AI would guess the missing leaf, but it would look like a generic, round blob, ignoring the specific jagged edges or unique lobes of the actual fossil.

So, they tried something bold: they unfroze the whole brain. They let the AI adjust its entire internal knowledge base, but very, very gently (using a tiny learning rate). This allowed the AI to adapt its general knowledge of shapes specifically to the weird, broken geometry of fossil leaves. It learned that a leaf isn't just a circle; it has specific curves, tips, and lobes.

The "Vein" Clue

There was another trick. The researchers realized that if they just asked the AI to guess the leaf shape, it might get the outline right but miss the details. So, they gave the AI a second job: draw the veins.

Think of it like this: if you ask someone to draw a car, they might get the wheels and the body right. But if you also ask them to draw the engine and the exhaust pipes, they are forced to understand how the whole machine fits together. By forcing the AI to predict the thin, tricky veins (which are often hidden or look like cracks in the rock), the model was forced to understand the structure of the leaf. This "structural prior" helped it draw a much more accurate outline of the hidden leaf parts.

From Fake to Real: The Synthetic Training

Here is the catch: there are no real photos of fossil leaves where scientists know exactly what the hidden part looks like. You can't take a picture of a leaf, bury it in rock, and then dig it up to see the "ground truth" of what was hidden.

To get around this, the team built a virtual training camp. They took 160 real, high-quality photos of clear leaves and used a 3D program to "break" them digitally. They simulated rocks covering parts of the leaves, added cracks, and even added weathering effects. They generated 11,000 of these fake fossil images to train the AI. The AI learned on these synthetic images, and then, miraculously, it worked on real fossils it had never seen before.

The Results: A Digital Time Machine

When they tested AmodalDINO on a validation set of synthetic images, it got the shape of the complete leaf right 95.0% of the time (measured by a metric called Dice) and the overlap with the true shape was 90.5%. That is incredibly high for a task involving guessing hidden information.

Even more impressive, when they tested it on 9 real fossil photographs (which had no "correct answer" to compare against), the model still produced plausible, scientifically useful results. It correctly extended the leaf blade into the rock, straightened out the main vein, and recognized that a leaf was lobed rather than just a simple oval.

Practical Magic: Running in Your Browser

The researchers didn't just stop at a powerful model; they made it practical. They shrunk the AI down using a technique called 4-bit quantization, which is like compressing a high-definition movie into a tiny file without losing the plot. This allowed the model to run entirely offline in a web browser.

They built an interactive demo where you can upload a photo of a fossil. The AI automatically finds the rock, cuts it out, and draws the four masks (visible, complete, main vein, fine veins). It even uses a ruler in the photo to calculate the actual size of the leaf in centimeters. For the general public, they added a "leaf revival" feature: using the AI's prediction, they can generate a colorful, living version of the ancient leaf, showing what it might have looked like when it was alive millions of years ago.

What It Can't Do (Yet)

The paper is honest about its limits. The model currently assumes there is only one leaf in the picture. If you show it a rock with two overlapping leaves, it will merge them into one giant, weird shape. It also struggles with very fine details if the rock covers too much of the leaf. And, because it was trained on synthetic data, there is still a small gap between its perfect digital training and the messy reality of real fossils.

But the core discovery is solid: by unfreezing the AI's brain and teaching it to look at the "skeleton" (the veins) alongside the "skin" (the leaf shape), we can finally teach computers to imagine the invisible parts of the past. It's a small step toward letting paleobotanists see the whole forest, even when they only have a single, broken leaf in their hand.

Technical Summary: Amodal Reconstruction of Leaf Fossil Images

Problem Formulation

Fossil leaves are rarely preserved in their entirety; sedimentary rock often obscures, fractures, or erodes the lamina. While paleobotany relies on the complete silhouette and venation patterns of leaves for species identification and climate reconstruction, standard semantic and instance segmentation methods only label the visible portions, leaving the missing tissue unrecovered.

This work addresses amodal reconstruction for leaf fossils: predicting the complete spatial extent of a leaf and its internal venation from a single RGB image where parts are hidden by rock. The task presents unique challenges compared to standard amodal segmentation (e.g., for cars or pedestrians):

Lack of rigid templates: Every fossil specimen has a unique organic shape, preventing the network from relying on a canonical silhouette.
Data scarcity: Real fossil images with pixel-level amodal and venation labels do not exist at the scale required for training dense-prediction transformers, necessitating a synthetic training dataset and bridging a domain gap.
Low-contrast structures: Thin veins are easily confused with cracks and mineral seams in the rock matrix.
No visible mask input: Unlike nearly all prior amodal methods that require a ground-truth or predicted visible mask as input to "grow" the object, this setting assumes no such mask is available. The boundary between the leaf and the rock is often ambiguous, and no reliable detector exists for "leaf embedded in rock."

Methodology: AmodalDINO

The proposed solution, AmodalDINO, is an end-to-end multi-head dense-prediction model that takes an RGB image (and an optional, coarse region-of-interest hint) and outputs four masks simultaneously:

Visible leaf tissue.
Complete amodal leaf (including hidden parts).
Amodal primary vein.
Fine detail veins.

Architecture and Design Choices

The model is built upon a DINOv3 ViT-L/16 backbone and a DPT decoder. The authors identify two critical design choices that deviate from standard protocols and enable the model to learn the structural shape priors of leaves:

Fully Fine-Tuning the Backbone: Contrary to the standard practice of freezing the pre-trained DINOv3 backbone and training only the decoder, AmodalDINO fully fine-tunes the backbone at a small learning rate ( $10^{-5}$ ). The authors argue that a frozen backbone only extrapolates a generic, low-frequency envelope. Unfreezing the layers allows the model to adapt features to the specific geometry of the specimen, snapping margins to the actual shape while preserving generalization capabilities.
Auxiliary Venation Heads: The model attaches auxiliary heads for primary and detail veins alongside the leaf heads. The authors posit that a coherent venation pattern can only be drawn on a coherent lamina. By supervising the midrib and veins, the shared trunk is forced to represent lobes and sinuses, providing a structural prior that improves the fidelity of the completed leaf shape. This also helps the model distinguish true veins from rock cracks.

Loss Function

The training objective employs a completion-oriented multi-head loss:

Boundary-weighted BCE: Emphasizes gradients on thin structures and margins.
Head-specific overlap terms: Uses soft Dice for visible/detail veins and a soft Tversky loss for completion heads (amodal leaf/veins) to penalize over-growth into stone more than missed pixels.
Completion Reweighting: The loss is up-weighted specifically in the occluded region (where the camera sees nothing) to prevent the model from simply reproducing the visible mask.
Soft Structural Constraints: Enforces that the visible mask lies within the amodal leaf, and both vein maps lie within the leaf.

Data Generation

Due to the lack of real labeled data, the authors constructed a synthetic dataset using the NMNS Cleared Leaf Database. They selected 160 diverse, high-quality leaves, segmented them, and traced their venation. Using Blender, these leaves were composited onto randomized stone models with varied poses, lighting, and textures. Breakage was simulated via post-processing in OpenCV, resulting in 11,000 training images. The model is trained exclusively on this synthetic data.

Results

Synthetic Validation

On the held-out synthetic validation set, AmodalDINO achieves:

95.0% Dice / 90.5% IoU for the amodal leaf.
98.5% Dice for the visible leaf.
65.4% Dice for the primary vein and 55.3% Dice for detail veins.
The gap between visible and amodal performance (8.0 IoU points) indicates the model is performing genuine hallucination of hidden tissue rather than mere recognition.

Transfer to Real Fossils

Despite being trained only on synthetic data, the model transfers effectively to real fossil specimens. Qualitative results show the model correctly extends the blade into the rock, recovers a coherent midrib, and distinguishes lobed leaves from simple ellipses.

Public Benchmarks (KINS and COCOA-cls)

To validate the generalizability of the approach, the authors stripped the model to two heads (visible and amodal leaf) and removed the ROI channel, testing it on standard amodal benchmarks without any visible mask input:

KINS: 85.05 full mIoU / 66.65 occluded mIoU (highest reported occluded mIoU in the table).
COCOA-cls: 80.90 full mIoU / 38.15 occluded mIoU.
The authors note that while the model is competitive on the occluded region, it trails behind methods like Amodal SAM on COCOA-cls, likely due to the lack of a strong category-specific shape prior in their "category-free" leaf setting.

Deployment and Quantization

The model was successfully deployed in two ways:

In-Browser: Quantized to 4-bit weights (reducing size from ~1.27 GB to ~269 MB) and running entirely offline via ONNX Runtime Web. It maintains an IoU of 0.910 compared to the FP32 reference on real fossils.
Local GPU Pipeline: Includes ruler-based calibration for surface area estimation and an optional generative visualization stage using a Flux.2 diffusion model to render "living" leaves based on the reconstructed masks.

Key Contributions

Problem Formulation: Defined the task of joint amodal reconstruction of leaf silhouettes and venation from a single RGB image without visible mask input.
Model Architecture (AmodalDINO): An end-to-end model requiring no upstream instance segmenter or visible mask, predicting visible and amodal regions jointly.
Design Insights: Identified that fully fine-tuning a self-supervised backbone (rather than freezing it) and using auxiliary structural heads (venation) are critical for learning shape priors in data-scarce, category-free scenarios.
Reproducible Recipe: Provided a complete pipeline including synthetic data generation, a two-rate learning schedule, and specific loss weighting.
Practical Deployment: Demonstrated a quantized, browser-based inference system with ruler-based calibration and generative visualization.

Significance and Limitations

The paper claims that the proposed method successfully recovers the "invisible" parts of leaf fossils, enabling paleobotanists to reason about complete organs from partial observations. The significance lies in demonstrating that a foundation model, when properly adapted (unfrozen backbone + structural supervision), can extrapolate complex organic shapes without rigid templates or visible masks.

The authors acknowledge several limitations:

Single-Leaf Assumption: The model implicitly assumes one leaf per image; it merges multiple overlapping specimens into a single silhouette, though it can still recognize distinct midribs.
Domain Gap: Training is entirely synthetic, and while transfer is good, fine venation under heavy occlusion remains unreliable.
Resolution: The fixed 448x448 resolution compresses original information.
Generative Visualization: The "living leaf" rendering is currently for illustration and may hallucinate details.

The authors conclude that the core findings—unfreezing the backbone and using auxiliary structural tasks—may be applicable to other amodal-related problems beyond paleobotany.