Digital Twin Driven Textile Classification and Foreign Object Recognition in Automated Sorting Systems

This paper presents a digital twin-driven dual-arm robotic system that integrates RGBD sensing, tactile feedback, and state-of-the-art Visual Language Models to achieve robust, real-time textile classification and foreign object detection for automated sustainable recycling.

The Gist

Imagine a chaotic laundry room where clothes are thrown into a giant, messy basket. Mixed in with your shirts and socks are random junk: a plastic water bottle, a soda can, or even a stray toy. Now, imagine a robot trying to sort this mess. This is the problem the authors of this paper are trying to solve.

Here is a simple breakdown of their solution, using some everyday analogies.

1. The Problem: The "Deformable" Nightmare

Sorting rigid boxes is easy for robots. But clothes? Clothes are like wet spaghetti. They flop, twist, hide parts of themselves, and get tangled with other items. If you throw a shirt and a pair of pants into a pile, they become a single, confusing blob.

Furthermore, in a recycling plant, you don't just want to sort clothes; you need to spot the "foreign objects" (like that plastic bottle) and throw them away. Old robots are bad at this because they only know exactly what they were programmed to see. If they see something new, they get confused.

2. The Solution: A "Digital Twin" and a "Super-Brain"

The team built a robotic system that uses two main tricks:

The Digital Twin (The Virtual Sandbox)
Think of a Digital Twin as a perfect video game copy of the real robot's world.

• How it works: Before the real robot moves its arm, it simulates the move in this virtual copy.
• The Benefit: It's like playing a flight simulator before flying a real plane. The robot checks, "If I reach for this shirt, will I crash into the basket or the table?" It plans a safe path in the virtual world first, ensuring the real robot doesn't bump into anything. It even creates a 3D map of the shirt it's holding so it knows exactly where to grab it next time.

The Visual Language Model (The Super-Brain)
Instead of using a simple camera that just says "I see a red blob," they used Visual Language Models (VLMs).

• The Analogy: Think of a standard robot camera as a toddler who can only point and say "Red" or "Blue." A VLM is like a smart adult who can read a book and look at a picture.
• How it works: You show the robot a picture of a sock and ask, "Is this a sock, a shirt, or a trash can?" The AI doesn't just match patterns; it understands the concept. It can say, "That looks like a sock," or "That's a soda can, not a sock." This allows the robot to handle clothes it has never seen before and spot foreign objects easily.

3. The Robot's Job: The "Pick, Shake, and Sort" Dance

The system uses two robot arms (named Alice and Bob) working together:

1. The Grab: Alice reaches into the messy basket. It uses special "fingertips" that can feel pressure (like human skin) to know if it actually grabbed something or just squeezed air.
2. The Shake: Once it grabs a shirt, it gives it a little shake. This is like you shaking out a wet towel to get the water off. It helps untangle the clothes so they don't get stuck together.
3. The Inspection: Alice lays the item flat on a table.
4. The Brain Scan: A camera takes a picture and sends it to the "Super-Brain" (the VLM). The AI looks at the image and shouts out the answer: "Shirt!" or "Soda Can!" or "Empty table!"
5. The Sort: Based on the answer, Alice moves the item to the correct bin.

4. The Results: Who Won the Race?

The researchers tested nine different "Super-Brains" (different AI models) to see which one was the best at this job.

• The Winner: The Qwen family of models was the champion. It was the most accurate, correctly identifying shirts, socks, and trash about 88% of the time. It was great at spotting foreign objects, which is crucial for recycling.
• The Speedster: The Gemma model was a bit less accurate but much faster. It's like a sprinter who might trip occasionally but gets to the finish line quickly. This is good for robots that need to move very fast.
• The Hallucination Problem: Some of the older or weaker models got "confused." They would look at an empty table and say, "I see a shirt!" (This is called hallucinating). The Qwen models were much better at saying, "I see nothing," when the table was empty.

5. Why Does This Matter?

This isn't just about robots folding laundry.

• Recycling: As the world tries to recycle more clothes, we need machines that can handle the mess without human help.
• Safety: The "Digital Twin" ensures the robot doesn't break itself or the factory equipment.
• The Future: This system proves that we can combine "feeling" (tactile sensors), "seeing" (cameras), and "thinking" (AI language models) to solve very messy, real-world problems.

In short: They built a robot that uses a virtual simulation to avoid crashing and a smart AI brain to understand what it's holding, making it possible to automatically sort messy piles of clothes and trash with high accuracy.

Technical Summary

Here is a detailed technical summary of the paper "Digital Twin–Driven Textile Classification and Foreign Object Recognition in Automated Sorting Systems."

1. Problem Statement

The paper addresses the critical challenges in sustainable textile recycling, specifically the automation of sorting deformable garments (clothing) from unsorted, cluttered heaps. Key difficulties include:

• Deformability: Garments have high intra-class variability, self-occlusion, and non-rigid dynamics, making them difficult for robots to grasp and manipulate.
• Clutter and Foreign Objects: Recycling streams often contain mixed piles with foreign objects (plastic, metal, packaging) that must be detected and separated.
• Regulatory Pressure: The upcoming EU Digital Product Passport (DPP) requires enhanced traceability and material transparency, necessitating robust perception systems for legacy and non-registered textiles.
• Limitations of Previous Work: Existing solutions often rely on pre-trained Convolutional Neural Networks (CNNs) with fixed classes or operate in controlled lab settings that lack scalability, robustness to large garments, and foreign object detection capabilities.

2. Methodology

The authors propose a dual-arm robotic cell integrated with a Digital Twin and Visual Language Models (VLMs) to create a robust, multi-stage sorting system.

A. Robotic Hardware & Setup

• Robots: Two UR7e manipulators (named "Alice" and "Bob") equipped with Robotiq 2F-140 grippers.
• Sensing:
  • Vision: Two Intel RealSense cameras (RGB-D) for grasp detection and object classification.
  • Tactile: CapTac capacitive finger sensors on Robot Alice to detect grasp success, normal/shear forces, and prevent gripper over-closure.
• Workflow:
  1. Pick: Robot Alice grasps an item from a cluttered basket (Zone A).
  2. Transfer & Spread: Alice moves the item to an inspection table (Zone B), performing a "shaking" motion to loosen the garment and pulling it over the edge to induce passive spreading for better visibility.
  3. Classify: A VLM analyzes the RGB image of the garment in Zone B.
  4. Sort: Based on classification, Alice moves the item to Zone C (dedicated bins or handover to Robot Bob).

B. Digital Twin & Motion Planning

• Environment: A virtual replica of the robotic cell is maintained using MoveIt within the RViz environment.
• Collision Avoidance: The system integrates real-time 3D point clouds of the inspected garments into the digital twin. This allows for collision-aware path planning, ensuring the robot avoids obstacles (including the garments themselves) during manipulation.
• Data Flow: ROS 2 (Jazzy) manages communication between the robots, sensors, and the digital twin. Point clouds are segmented and can be exported as .stl files for simulation or further training.

C. Classification Engine (VLMs)

Instead of traditional CNNs, the system utilizes Visual Language Models (VLMs) for zero-shot, semantic classification.

• Classes: Trousers, Shirt, Underwear, Sock, Other (foreign objects/unidentifiable), and Empty.
• Implementation: Models run via the Ollama Python API. The system uses a prompt engineering strategy to force single-word outputs to ensure parsing reliability.
• Benchmarking: Nine VLMs from five families were tested:
  • Qwen (Qwen3-VL, Qwen3.5)
  • Llama (Llama 3.2, Llama 4)
  • Gemma (Gemma 3)
  • LLaVA
  • MiniCPM

3. Key Contributions

1. Integrated Robotic System: A novel architecture combining VLM-based semantic reasoning with conventional grasp detection and tactile feedback in a dual-robot industrial setting.
2. Digital Twin for Manipulation: The integration of segmented 3D point clouds of deformable objects into the digital twin for collision-aware path planning, significantly improving manipulation reliability in cluttered environments.
3. Comprehensive VLM Benchmark: A rigorous evaluation of nine state-of-the-art VLMs on a dataset of 223 inspection scenarios, specifically assessing:
   • Per-class accuracy for specific garment types.
   • Foreign object detection capabilities.
   • Hallucination behavior (e.g., detecting objects when the table is empty).
   • Computational performance under hardware constraints.
4. Open Data: The authors commit to releasing the raw imagery and labels of all processed objects for future research.

4. Experimental Results

The system was tested with 219 items (garments + foreign objects) plus 4 empty scenarios.

• Accuracy:
  • The Qwen model family achieved the highest overall accuracy. Qwen3-VL:235b and Qwen3.5:35b both reached ~87.9% overall accuracy.
  • Qwen models showed superior performance in detecting "Other" (foreign objects) and specific classes like socks (100% accuracy for Qwen3-VL:235b).
  • Gemma3:12b offered a competitive speed-accuracy trade-off (76.23% overall) suitable for edge deployment but struggled with larger garments due to placement issues.
  • Llama models exhibited higher hallucination rates, often failing to detect empty tables or misclassifying items after multiple similar instances.
• Hallucination & Robustness:
  • Gemma3:12b was the only model to correctly identify 100% of empty table scenarios.
  • LLaVA frequently violated output constraints by returning full sentences instead of single-word classes.
• Computational Performance:
  • Lightweight models (Gemma3, MiniCPM): Fast inference (~0.4s – 0.6s), suitable for real-time edge deployment.
  • Large models (Qwen3-VL:235b, Llama4): Slower inference (up to 20s for Qwen3.5:122b on H200) but higher accuracy.
  • Hardware: While large models require high-end GPUs (e.g., Nvidia H200 with 144GB VRAM), the Qwen3.5:35b model demonstrated that high accuracy is achievable on high-end consumer GPUs.

5. Significance and Outlook

• Industrial Feasibility: The study proves that combining semantic VLM reasoning with traditional robotics (grasp planning, digital twins) is feasible for scalable, autonomous textile sorting in realistic, cluttered industrial environments.
• Regulatory Compliance: The system directly addresses the needs of the EU Digital Product Passport by enabling the classification of legacy textiles and foreign objects without pre-training on specific classes.
• Future Work: The authors plan to implement robot-to-robot handovers, use multi-arm spreading for better garment flattening (to improve classification accuracy for large items), and explore ensemble methods (combining Qwen and Gemma) to balance speed, accuracy, and hallucination reduction.

In conclusion, this work represents a significant step forward in deformable object manipulation, moving beyond rigid lab setups to a robust, perception-driven system capable of handling the complexities of real-world textile recycling.