💻 computer science

Adversarial robustness and confidence miscalibration in emotion-based HRI systems

This paper demonstrates that facial emotion recognition systems for robotics, while achieving high clean-set accuracy, suffer from catastrophic and structurally uncalibratable confidence miscalibration under adversarial attacks, rendering them fundamentally unsafe for physical control despite appearing highly confident.

Original authors: Ezekiel Olayide Tolulope, Franklin Tchakounté, Derrick Méthode BAGAZA, Shanay Rab, Almas Baimagambetov

Published 2026-10-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ezekiel Olayide Tolulope, Franklin Tchakounté, Derrick Méthode BAGAZA, Shanay Rab, Almas Baimagambetov

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a robot designed to help people, perhaps in a hospital or a school, that is supposed to understand human feelings by looking at faces. If the robot sees a smile, it might offer a cheerful greeting; if it sees a frown, it might offer comfort. This ability to read emotions is a cornerstone of modern social robotics, relying on computer programs trained to recognize facial expressions. For these machines to be safe and useful, their creators must trust that the robot's interpretation of a face is correct. If the robot misreads a neutral face as angry, it might react with unnecessary caution or aggression, potentially causing harm or distress. The core question for engineers is not just whether the robot can recognize emotions accurately in a perfect, controlled setting, but whether it can be trusted to make decisions when the world is messy or when someone tries to trick it.

A team of researchers set out to test exactly this trustworthiness. They examined five different computer systems designed to recognize five basic emotions: anger, happiness, sadness, surprise, and a neutral state. These systems were put through a series of rigorous tests to see how they performed when the images they received were slightly blurry, dimly lit, or covered in digital noise. More importantly, the researchers subjected these systems to a specific type of digital trickery known as an adversarial attack. In this scenario, a computer program makes tiny, almost invisible changes to a photo of a face—changes so small a human eye would never notice them—but these changes are calculated to confuse the robot completely. The goal was to see if the robot would still recognize the emotion correctly, and crucially, whether the robot would know that it was confused.

The results were stark and revealed a serious flaw in how these systems are currently built. When the researchers introduced even the smallest amount of digital trickery, the robots' ability to recognize emotions collapsed. Systems that were previously correct about 76 percent of the time in clean conditions dropped to near zero accuracy. They began to guess wildly, often identifying a happy face as angry or a sad face as surprised. The most alarming discovery, however, was not that the robots made mistakes, but that they were completely unaware of their own errors. Even when the robots were being tricked into seeing the wrong emotion, they remained incredibly confident in their wrong answers. In fact, the computer programs reported being nearly 96 percent sure of their incorrect guesses.

This finding challenges a common safety strategy used in robotics. Engineers often program robots to ignore a prediction if the computer's confidence score is too low, assuming that a low score means the robot is unsure and should wait for a better reading. The researchers found that this safety net is useless against these digital tricks. Because the robots remained so confident even when they were wrong, the safety gate never closed. The robots proceeded to act on their false beliefs with total certainty. The team proved mathematically that simply adjusting the computer's internal settings to make it more cautious would not fix the problem. The way these systems calculate confidence is structurally incapable of distinguishing between a real face and a tricked one when the trick is strong enough.

The researchers concluded that relying on a single snapshot of a face and the computer's confidence score is not enough to keep a robot safe. They demonstrated that a robot's ability to recognize an emotion and its ability to safely act on that emotion are two different things. A system can be very good at recognizing faces in a lab but still be dangerously fragile in the real world. To build truly secure robots, the researchers argue that we need a new layer of verification. Instead of letting the robot act immediately after seeing a face, the system should check the prediction against other evidence, such as how long the expression has lasted or whether the robot's other sensors agree with the visual input. Until such safeguards are in place, the robots that rely on reading our faces may be more likely to misunderstand us than to understand us, especially when faced with even the slightest attempt to deceive them.

Technical Summary: Adversarial Robustness and Confidence Miscalibration in Emotion-based HRI Systems

Problem Statement

The paper addresses a critical security gap in Human-Robot Interaction (HRI) systems that utilize Facial Emotion Recognition (FER). While FER models are typically evaluated based on clean-set recognition accuracy (e.g., precision, recall, F1-score), these metrics fail to assess whether a prediction is trustworthy enough to trigger physical robot actions. The core problem is that a classifier may be accurate on standard data yet brittle under environmental degradation or adversarial perturbations. Crucially, the authors investigate whether confidence thresholding—a common mitigation strategy where the system acts only if the maximum softmax probability exceeds a threshold—can reliably distinguish between legitimate and adversarial inputs. The study posits that if adversarial attacks produce high-confidence errors, confidence-based gating fails precisely when security mechanisms are most needed.

Methodology

The authors conducted a controlled empirical study using a five-class subset of the FER2013 dataset (Angry, Happy, Neutral, Sad, Surprise).

Pipelines Evaluated: Five FER pipelines were compared:
1. DeepFace: An off-the-shelf baseline (zero-shot).
2. Strategy 1: A regularized custom CNN trained from scratch.
3. Strategy 2: VGG16 with a frozen convolutional base and trained classification head.
4. Strategy 3: VGG16 fully retrained (fine-tuned end-to-end).
5. Strategy 4: A deep residual CNN trained from scratch.
Evaluation Environment: All experiments were conducted on CPU-only hardware to reflect real-world robot deployment constraints.
Stress Conditions:
- Environmental Corruption: Gaussian noise, blur, brightness reduction, and JPEG compression at five severity levels.
- Adversarial Attacks: Untargeted FGSM (Fast Gradient Sign Method) and PGD (Projected Gradient Descent) attacks across a perturbation budget ( $\epsilon$ ) sweep.
Calibration Analysis: Post-hoc temperature scaling was applied to logits to test if recalibration could separate clean from adversarial inputs. The operational gate was fixed at a threshold $\tau = 0.70$ .
Metrics: The study moved beyond standard accuracy to measure actuation soundness (the risk of a wrong action given a prediction is admitted) and detector performance (Youden index, AUROC) of the confidence gate.

Key Contributions

Formalization of the Perception-to-Action Gate: The paper defines "actuation soundness" as a property of the admitted subpopulation, distinguishing it from general recognition accuracy. It establishes that a system can be accurate on average but systematically wrong on the specific subset of inputs it chooses to act upon.
Cross-Architecture Adversarial Evaluation: The study demonstrates that adversarial fragility is a shared property across diverse architectures (custom CNNs, transfer learning, and residual networks) under a common perturbation budget.
Impossibility Result for Global Calibration: The authors prove analytically that global temperature scaling cannot repair the failure of confidence gates under adversarial attack. Because the maximum softmax probability is strictly increasing with the logit-gap vector, and temperature scaling applies an identical transformation to all inputs, no single temperature can simultaneously admit legitimate predictions (with smaller gaps) and reject adversarial ones (with larger gaps) when the adversarial gap dominates.
Detector Bounds: The paper derives bounds showing that the Youden index of the confidence gate is constrained by the adversarial admission rate, proving that the gate is structurally incapable of separating clean and adversarial inputs under the observed conditions.

Results

Clean Performance: The deep residual CNN (Strategy 4) achieved the highest clean-set accuracy (0.760) and macro-F1 (0.753) at real-time CPU throughput.
Environmental Robustness: Environmental corruption caused performance degradation proportional to severity. However, adversarial attacks caused a uniform collapse.
- FGSM: Reduced all models to below 0.16 accuracy at $\epsilon = 0.01$ .
- PGD: Reduced all models to near-zero accuracy (effectively <0.005) at $\epsilon \ge 0.02$ .
Confidence Miscalibration: Despite the collapse in accuracy, attacked models remained highly confident. Under PGD, the mean confidence of adversarial predictions reached 0.964.
Failure of Thresholding:
- 96.0% of adversarial predictions exceeded the operational threshold ( $\tau = 0.70$ ) even after temperature scaling.
- The confidence gate's AUROC was 0.144, indicating the detector is inverted (it admits adversarial inputs more often than legitimate ones).
- The Youden index was measured at -0.261, confirming the gate is anti-correlated with the need for rejection.
Calibration Invariance: Temperature scaling (fitted at $T=0.982$ ) had negligible effect on the separation of clean and adversarial inputs. Theoretical analysis confirmed that the ROC curve of the max-softmax score is invariant to temperature scaling under the assumption of ordered runner-up structures.

Significance and Claims

The paper claims that adversarial fragility in FER is not architecture-specific and that high-confidence errors are a structural feature of the attack, not a calibration artifact that can be fixed by rescaling.

Separation of Concerns: The authors argue that recognition accuracy and actuation soundness are distinct properties. High clean-set accuracy does not guarantee safe actuation.
Limitation of Current Mitigations: The study concludes that relying solely on confidence thresholding is insufficient for security-critical HRI. The "confidence gate" fails because adversarial attacks concentrate errors in the high-confidence region that the gate is designed to pass.
Architectural Implication: The paper motivates the necessity of a system-level verification layer between the FER module and the robot's action policy. This layer must integrate confidence with other evidence streams (e.g., temporal persistence, action-rate constraints) rather than relying on instantaneous softmax scores.
Scope: The authors explicitly state they do not propose new robust training methods or better detectors. Instead, they demonstrate that the most commonly deployed decision rule (confidence thresholding) fails under adversarial conditions and that this failure cannot be remedied by post-hoc calibration. The findings suggest that safe emotion-guided robotics requires explicit verification stages that extend beyond the classifier's internal confidence scores.