MICA: Multi-Agent Industrial Coordination Assistant

This paper introduces MICA, a privacy-preserving, speech-interactive multi-agent system that leverages Adaptive Step Fusion and a safety-audited coordination topology to deliver robust, real-time industrial assistance for assembly and maintenance tasks on resource-constrained hardware.

The Gist

Imagine you are working on a complex assembly line, like building a high-tech engine or a sophisticated piece of machinery. You aren't a master engineer; you're a skilled worker who needs to assemble parts quickly, but the instructions are long, the tools are specific, and one wrong move could break the machine or hurt someone.

In the past, you might have had a manual (a paper book) or a cloud-based app that needed Wi-Fi. But what if the factory has no internet, the data is too sensitive to send to the cloud, or the instructions change every day?

Enter MICA (Multi-Agent Industrial Coordination Assistant). Think of MICA not as a single robot, but as a highly organized, on-site "Brain Trust" that lives entirely on a small computer you can wear or carry.

Here is how MICA works, broken down into simple concepts:

1. The "Eyes" (Depth-Guided Vision)

Imagine you are wearing smart glasses. Most cameras just see a blurry mess of shapes. MICA's "eyes" are special. They use depth perception (like human eyes) to figure out exactly what you are looking at and how close it is.

• The Analogy: If you are holding a wrench near a bolt, a normal camera might see "metal." MICA sees "The worker is holding a wrench right next to the bolt." It filters out the background noise so it only focuses on the parts that matter right now.

2. The "Step Detective" (Adaptive Step Fusion)

This is MICA's superpower. It has to guess which step of the assembly you are on.

• The Problem: Sometimes the camera gets blocked, or the lighting is bad. A standard computer might get confused and say, "You're on Step 5!" when you are actually on Step 3.
• The Solution: MICA uses two detectives working together:
  1. The Rule-Follower: It checks a digital checklist (the "State Graph") to see if the parts you have should be there.
  2. The Visual Matcher: It looks at a photo gallery of what each step looks like and finds the closest match.
• The Magic: These two detectives vote. But if they disagree, MICA doesn't just guess. It asks you (the worker) for help via voice. If you say, "No, I'm actually on Step 4," MICA instantly learns from that correction and updates its internal logic. It's like a GPS that learns from your driving habits to give you better directions next time.

3. The "Brain Trust" (Multi-Agent System)

Instead of one giant AI trying to do everything (which often gets confused), MICA splits the job into five specialized experts who talk to each other:

• The Assembly Guide: Knows how to put things together.
• The Parts Advisor: Knows what every screw and gear is called.
• The Maintenance Expert: Knows how to fix broken things.
• The Fault Handler: Knows what to do when things go wrong.
• The Safety Inspector: The most important one. This agent acts like a strict referee. It listens to everything the other experts say and checks it against the safety rules. If the "Assembly Guide" suggests a dangerous move, the "Safety Inspector" slams the brakes and says, "No, that's unsafe!" before you hear it.

4. The "Conversation" (Speech Interaction)

You don't need to type or touch a screen with greasy hands. You just talk.

• You say: "How do I tighten this bolt?" or "Is this part broken?"
• MICA listens: It understands your voice, figures out which expert you need, gets the answer, checks it for safety, and speaks back to you instantly.
• The Loop: If MICA gets it wrong, you can say, "That's not right," and it fixes its answer immediately.

Why is this a big deal?

Most AI assistants today need the internet (the Cloud) to work. They send your factory data to a server, get an answer, and send it back.

• The Problem: Factories often have bad internet, and companies don't want to share their secret blueprints with a cloud server.
• MICA's Win: It runs offline on a small device. It keeps your data private, it works even if the Wi-Fi dies, and it's fast enough to keep up with the speed of the assembly line.

The "Benchmark" (The Report Card)

The researchers didn't just build it; they tested it against other ways of organizing AI teams. They created a "test track" with four different team structures:

1. The Huddle: Everyone talks to everyone (slow, messy).
2. The Boss: One leader tells everyone what to do (fast, but if the boss is wrong, everyone is wrong).
3. The Assembly Line: Information passes from person to person (efficient, but errors pile up).
4. The Debate Club: Everyone argues and votes (accurate, but takes too long).

MICA beat all of them. It was the fastest, used the least energy (battery), and gave the most accurate, safe answers. It found the perfect balance between being smart, being fast, and being safe.

In a Nutshell

MICA is like having a team of five expert mechanics, a safety inspector, and a personal trainer all living in your smart glasses. They watch what you do, listen to what you say, learn from your mistakes, and guide you through complex tasks without ever needing to connect to the internet. It makes high-tech manufacturing safer, faster, and easier for everyone.

Technical Summary

Here is a detailed technical summary of the paper "MICA: Multi-Agent Industrial Coordination Assistant".

1. Problem Statement

Modern manufacturing environments face significant challenges in providing real-time, adaptive assistance to workers. Key issues include:

• Complex Workflows: Assembly procedures are long-horizon, interdependent, and prone to errors by non-expert or rotating workers, leading to safety and quality costs.
• Constraints: Strict privacy requirements and limited connectivity often prevent cloud-based offloading. Additionally, collecting large annotated datasets for training is difficult due to confidentiality.
• Limitations of Existing Solutions: Current vision-based systems struggle with robust step understanding under occlusion and noise. Existing multi-agent Large Language Model (LLM) approaches are largely text-centric, simulated, or lack grounding in real-world sensor data, leading to unreliable coordination under partial observations.

The goal is to create a privacy-preserving, fully offline, edge-deployable assistant that combines egocentric vision, speech interaction, and multi-agent reasoning to guide assembly, troubleshooting, and maintenance.

2. Methodology

MICA is a perception-grounded system running entirely on edge hardware. It integrates three core modules:

A. Depth-guided Object Context Extraction

To handle dynamic assembly conditions and occlusion:

• Detection: Uses YOLOv11 trained on the Gear8 dataset to detect components.
• Stabilization: Aggregates detections over a sliding window ($L$ frames) and fuses bounding boxes using confidence-weighted averaging.
• Depth Refinement: Employs Depth-Anything to estimate pixel-wise depth. The system identifies the worker's primary focus (nearest component) and includes peripheral components within specific spatial and depth thresholds to capture context.

B. Adaptive Step Recognition (ASF)

This module resolves step ambiguities by fusing two complementary detectors and adapting online via user speech feedback:

1. State-Graph Detector: Uses workflow constraints derived from a Knowledge Base (KB) to score steps based on required components and their multiplicities. It enforces structural consistency.
2. Retrieval Detector: Compares the current video frame against a gallery of reference states using image embeddings (cosine similarity) to handle occlusion and detection noise.
3. Adaptive Step Fusion (ASF):
   • Fusion: Dynamically blends scores from both detectors using learnable weights and biases. It incorporates workflow compatibility and penalizes "jumping" steps (non-sequential transitions).
   • Online Adaptation: Uses natural language feedback from workers to update fusion weights without backpropagation. It employs a focal-style impact mechanism and anti-collapse regularization (scaling step sizes based on feedback frequency) to prevent the system from locking into a single incorrect class.

C. MICA-core (Multi-Agent Collaborative Reasoning)

A modular reasoning layer built on an instruction-tuned LLM (Qwen2.5-7B-Instruct):

• Routing: A lightweight router dynamically assigns queries to one of five specialized agents: Assembly Guide, Parts Advisor, Maintenance Advisor, Fault Handler, and a General Agent.
• RAG Paradigm: Agents retrieve evidence from a structured, local Knowledge Base (derived from manuals and component captures) to generate responses.
• Safety Auditing: A dedicated safety checker audits all outputs against rule-based constraints (e.g., correct tool usage, assembly order) to prevent unsafe recommendations before they reach the user.

D. Interactive Feedback Loop

The system uses Speech-to-Text (STT) and Text-to-Speech (TTS) for interaction. Crucially, workers can verbally confirm or correct step predictions in real-time, directly driving the online adaptation of the ASF module.

3. Key Contributions

1. MICA System: A fully offline, perception-grounded industrial assistant unifying egocentric vision, speech I/O, and role-specialized multi-agent reasoning on edge devices.
2. Adaptive Step Fusion (ASF): A novel mechanism that blends rule-based workflow constraints with retrieval-based visual similarity, enabling real-time correction through natural language feedback without heavy retraining.
3. New Benchmark & Metrics:
   • Established a controlled benchmark comparing four multi-agent topologies (SharedMemory, CentralizedBroadcast, HierarchicalPipeline, DebateVoting) under identical tools and knowledge access.
   • Introduced Knowledge Base Alignment (KBA) to measure factual consistency with the curated KB.
   • Introduced Energy per Successful Answer (E/succ) to evaluate energy-efficiency in real-time use.

4. Experimental Results

Experiments were conducted on the Gear8 dataset across five task categories (General, Assembly, Part Attributes, Maintenance, Fault Handling).

• ASF Performance: Online adaptation significantly improved robustness. For example, in Step S4 (where the baseline failed completely), ASF improved accuracy from 0% to 95.34% and F1-score by +90.36 after just 10 updates. It also reduced Expected Calibration Error (ECE) across all steps.
• Multi-Agent Benchmark:
  • MICA-core achieved the highest Task Success (63.13%) and Knowledge Base Alignment (19.12%).
  • It demonstrated superior efficiency with the lowest Latency (0.71s) and Energy per Successful Answer (2.05 kJ).
  • Baselines:
    • DebateVoting excelled in lexical overlap (BLEU/ROUGE) for simple attribute queries but suffered from high latency and energy costs due to redundant agent activations.
    • CentralizedBroadcast performed well on assembly tasks but consumed more energy.
    • HierarchicalPipeline was brittle; upstream errors propagated downstream without correction.
• Qualitative Analysis: MICA successfully reformulated canonical KB sequences into user-oriented lists and handled perception-only queries (e.g., duplicate checks) where baselines failed due to over-reliance on retrieval.

5. Significance

• Deployability: MICA demonstrates that complex, safety-critical industrial assistance can run entirely on edge hardware without cloud dependency, addressing privacy and connectivity constraints.
• Robustness: The ASF mechanism proves that lightweight, feedback-driven adaptation can correct systematic failures in step recognition, making the system adaptable to new procedures without massive re-annotation.
• Evaluation Standard: The paper shifts the evaluation paradigm from purely text-based metrics (BLEU/ROUGE) to grounded metrics (KBA) and efficiency metrics (Energy/Latency), highlighting that in industrial settings, factual accuracy and safety are more critical than lexical fluency.
• Future Impact: MICA provides a blueprint for the next generation of privacy-preserving, adaptive industrial assistants capable of operating in dynamic, real-world factory environments.