🤖 machine learning

What's on My Network? Using Large Language Models to Identify Real-World IoT Devices at Scale

This paper introduces a scalable, instruction-tuned LLM pipeline that reframes IoT device identification as a language modeling task, achieving high accuracy and robustness across 2,015 vendors by leveraging a newly curated, high-fidelity dataset and addressing challenges like sparse metadata and adversarial manipulation.

Original authors: Rameen Mahmood, Tousif Ahmed, Sai Teja Peddinti, Danny Yuxing Huang

Published 2026-07-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Rameen Mahmood, Tousif Ahmed, Sai Teja Peddinti, Danny Yuxing Huang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you walk into a crowded room where everyone is wearing a mask, and some masks are even upside down or painted over. You can't see their faces, but you can hear them talking, see what they are carrying, and notice who they are whispering to. Your goal? To figure out exactly who everyone is in that room.

This is the problem the researchers in this paper are trying to solve, but instead of a room, it's your home Wi-Fi network, and instead of people, it's IoT devices (smart bulbs, cameras, thermostats, etc.).

Here is a simple breakdown of what they did, using everyday analogies:

The Problem: The "Ghost" Devices

In modern homes and offices, there are dozens of devices connected to the internet. Many of them don't have names, or they lie about who they are.

The "Silent" Device: Some devices don't announce themselves.
The "Imposter": Some devices use generic names like "Smart-Plug-123" instead of "Ring Camera."
The "Chameleon": Some change their identity to hide.

Security experts and homeowners usually try to identify these devices by looking at their "ID cards" (like MAC addresses or network names). But in the real world, these ID cards are often missing, blurry, or forged. It's like trying to identify a suspect in a lineup where half the photos are pixelated and the other half are wearing wigs.

The Solution: The "Super Detective" (LLM)

The researchers decided to stop treating this like a math problem (matching patterns) and start treating it like a language problem. They used a Large Language Model (LLM)—a type of AI that is really good at reading, understanding context, and connecting dots.

Think of the AI as a super-detective who has read millions of books about technology. Even if a device gives a vague clue (like "I talk to a server named tuya-cloud.com"), the detective knows, "Ah, tuya is a company that makes smart plugs for many different brands. So this is likely a smart plug."

How They Trained the Detective (The Two-Stage Pipeline)

You can't just ask a smart AI to guess on a messy dataset; it might get confused. So, they built a two-step training process:

Stage 1: The "Panel of Experts" (Pseudo-Labeling)
First, they didn't trust just one AI. They asked three different, very powerful AI models (LLaMA, GPT-4o, and Gemini) to look at the messy data and guess the device's brand.

The Analogy: Imagine a jury of three experts. If two say "It's an Amazon Echo" and one says "It's a generic speaker," the jury votes for "Amazon Echo."
They also used a special scoring system to decide which expert's opinion mattered most based on how much evidence they had.
Result: This created a "Gold Standard" set of labels (names) for the devices, even though no human had labeled them all.

Stage 2: The "Apprentice" (Instruction Tuning)
Now, they took a smaller, faster AI model (LLaMA 3.1 8B) and taught it using the "Gold Standard" labels from Stage 1.

The Analogy: This is like a master chef (the big jury) teaching a junior chef (the small model) how to cook. The master chef doesn't just say "make soup"; they explain why ("add salt because the broth is bland").
Curriculum Learning: They taught the junior chef in stages. First, they gave it easy cases (devices with clear names). Once the chef got good at those, they gave it harder, messier cases (devices with missing info or fake names). This helped the chef learn to handle the "real world" chaos.

The Results: How Good is the Detective?

The results were impressive:

Accuracy: The model correctly identified the brand of the device 98.69% of the time across over 2,000 different brands.
Handling Lies: Even when someone tried to trick the system by changing the device's name or hiding its identity, the model was still very good at figuring it out because it looked at all the clues together, not just one.
The "Long Tail": It didn't just get the famous brands (like Apple or Samsung) right; it also figured out obscure, rare brands that other systems completely missed.

Why This Matters

Currently, if you rent an Airbnb or stay in a hotel, you have no way of knowing if there is a hidden camera or a listening device on the network. You just have to trust that everything is safe.

This system acts like a security scanner that can look at the "digital noise" of your network and say, "Hey, that device claiming to be a smart light is actually a hidden camera from Brand X." It works even when the device tries to hide, and it works without needing to hack into the device or see its secret data.

The Catch (Limitations)

The paper is honest about what this system can't do yet:

Encryption: If the internet traffic is heavily encrypted (like using special privacy tools), some of the clues the detective relies on disappear, making it harder to guess.
Cost: Running this "super detective" takes more computer power than a simple checklist, so it's better for checking your network once a day or week, rather than checking every single second.
Hallucinations: Sometimes, if the clues are very weak, the AI might make a "smart guess" that turns out to be wrong (like guessing a device is made by Intel when it's actually just using an Intel chip).

In a Nutshell

The researchers built a system that turns the messy, confusing language of computer networks into a clear list of who is actually on your Wi-Fi. By teaching an AI to "read" the network like a story rather than just matching code, they created a tool that is much better at spotting hidden or disguised devices than anything we've had before.

Technical Summary: What's on My Network? Using Large Language Models to Identify Real-World IoT Devices at Scale

Problem Statement
The proliferation of IoT devices in shared environments (homes, enterprises, rentals) has outpaced the ability to identify them, creating significant risks to privacy, safety, and accountability. Existing identification methods face fundamental limitations in open-world settings:

Active probing (e.g., mDNS, Nmap) often fails because devices suppress broadcasts.
Passive observation relies on indirect signals (MAC OUIs, DHCP hostnames, DNS queries) that are frequently sparse, noisy, spoofed, or randomized.
Closed-world assumptions in prior machine learning approaches (statistical classifiers, deep learning) assume complete metadata and known device inventories, failing to handle the "long tail" of vendors, firmware variants, and anomalous states found in real deployments.
Deterministic matching (e.g., Fingerbank) struggles with aliasing (e.g., "Nest" vs. "Google") and indirect signals (e.g., OUIs pointing to chip manufacturers rather than consumer brands), achieving low accuracy (~28%) on real-world data.

The core challenge is to identify device vendors from fragmented, noisy, and potentially adversarial network metadata without relying on active cooperation or a pre-defined closed set of devices.

Methodology
The authors propose a two-stage semantic inference pipeline that reframes device identification as a language modeling task over semi-structured network metadata.

1. Dataset and Preprocessing
The system utilizes the IoT Inspector dataset, a crowdsourced corpus of 2.68M flow-level entries from over 6,000 homes. The preprocessing pipeline transforms raw flow logs into 216,000 unique device profiles by:

Canonicalizing domains and merging semantically equivalent entries.
Extracting persistent identifiers from mDNS/SSDP broadcasts while removing volatile fields (e.g., serial numbers).
Normalizing User-Agent strings and aggregating flows into canonical signatures.
Retaining missing values to simulate real-world sparsity during training.

2. Stage 1: High-Fidelity Pseudo-Label Generation
Since over half the dataset lacks user-provided labels, the authors construct a supervision set using an LLM ensemble:

Models: LLaMA 3.1 70B, GPT-4o, and Gemini 1.5 Pro.
Prompting: A "Joint + Chain-of-Thought" (CoT) strategy is used to generate structured outputs (Device Type, Vendor) with explicit reasoning.
Consolidation: Predictions are aggregated via Proxy Conditional Mutual Information (Proxy CMI) weighted voting. This metric quantifies the reliability of each input feature (e.g., OUI vs. hostname) for a specific model, allowing the system to select the most informative prediction when fields disagree.
Normalization: Synonymous aliases (e.g., "Nest" $\to$ "Alphabet Inc.") are normalized using Wikidata to reduce label fragmentation.
Outcome: This stage produces 216,000 high-confidence pseudo-labels across 2,015 vendors.

3. Stage 2: Instruction-Tuning for Deployment
The pseudo-labels serve as structured supervision to train a compact, deployable classifier:

Model: A quantized LLaMA 3.1 8B model (using QLoRA).
Curriculum Learning: Training occurs in two phases to ensure robustness:
- Phase I: Trained on a 35K "high-signal" subset (devices with clear hostnames) to establish unambiguous mappings.
- Phase II: Fine-tuned on the full 216K corpus, exposing the model to long-tail vendors, sparse metadata, and noisy inputs.
Loss Masking: Supervision is restricted to the vendor label span only, preventing the model from learning spurious patterns from its own generated rationales.

Key Contributions

Semantic Inference Pipeline: The first application of instruction-tuned LLMs to IoT device identification, treating network metadata as structured context for reasoning rather than fixed feature vectors.
Proxy CMI Framework: A model-agnostic, information-theoretic method for ranking feature reliability and consolidating ensemble predictions without requiring ground-truth labels.
Curriculum Learning Strategy: A two-phase training approach that successfully balances high-precision learning with generalization to sparse, long-tail, and adversarial inputs.
Open-Source Deployment: The model is integrated into a scalable, open-source application capable of processing residential network traffic in real-time.

Results

Accuracy: The final model achieves 98.69% Top-1 accuracy and 90.73% macro accuracy across 2,015 vendors on internal hold-out sets.
Generalization: On the external Mon(IoT)r Testbed (unseen data with temporal, geographic, and protocol drift), the model maintains 92.95% accuracy, significantly outperforming baselines like SVM (60.17%), XGBoost (67.31%), and Fingerbank (25.90%).
Long-Tail Performance: The curriculum learning strategy substantially improves performance on rare vendors (Tail $\le$ 10 samples), increasing accuracy from 95.70% (no curriculum) to 97.74%.
Robustness: The model demonstrates resilience to:
- Missing Fields: Performance degrades gracefully when key features (e.g., OUI, hostname) are masked.
- Adversarial Manipulation: It resists spoofed hostnames and misleading user labels by grounding predictions in cross-field consistency.
- Open-Set Conditions: It correctly identifies 89.4% of devices belonging to vendors entirely absent from the training data.
Efficiency: The fine-tuned 8B model achieves 5.29 devices/second with 0.19s latency on a single GPU, offering a viable trade-off between cost and the superior accuracy of LLMs compared to lightweight baselines.

Significance and Claims
The paper claims that instruction-tuned LLMs provide a scalable, interpretable, and robust foundation for trustworthy device identification in open-world IoT environments. By shifting from closed-world fingerprinting to semantic inference, the approach addresses the structural limitations of prior systems that rely on deterministic pattern matching. The system successfully reconciles fragmented metadata, infers vendors absent from training data, and remains robust against missing fields and adversarial manipulation. The authors position this work as a step toward transparent network monitoring, enabling users and administrators to answer the foundational question, "What's on my network?" even in the absence of perfect metadata.