💻 computer science

Security Barriers to Trustworthy AI-Driven Cyber Threat Intelligence in Finance: Evidence from Practitioners

This mixed-methods study identifies four key socio-technical barriers—shadow AI use, poor operational integration, attacker-perception gaps, and inadequate model security—that hinder the trustworthy deployment of AI-driven Cyber Threat Intelligence in finance, while proposing operational safeguards to address these challenges.

Original authors: Emir Karaosman, Advije Rizvani, Irdin Pekaric

Published 2026-03-25

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Emir Karaosman, Advije Rizvani, Irdin Pekaric

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the financial world (banks, investment firms, etc.) as a massive, high-security fortress. For years, the guards (security analysts) have used a set of rigid, pre-written rulebooks to spot intruders. "If a door opens at 3 AM, sound the alarm." It works, but it's slow and easily tricked by clever thieves who know how to slip past the rules.

Enter Artificial Intelligence (AI). The promise is that AI is a super-smart, super-fast guard dog that can sniff out thieves before they even touch the door, predicting their moves based on patterns humans can't see.

This paper is like a report from the people actually living inside the fortress. They asked: "We bought these super-dogs, but why aren't we using them to guard the gates yet?"

Here is the breakdown of their findings, using simple analogies:

1. The "Shadow Dog" Problem (Shadow AI)

The Issue: The bank has strict rules: "No outside dogs allowed." But the guards are so overwhelmed that they secretly bring in their own personal, untrained dogs from the internet (public AI tools) to help them catch a few extra thieves.
The Risk: These "shadow dogs" aren't vetted. They might bark at the wrong things, or worse, they might accidentally tell the thieves where the gold is kept because the guard typed a secret password into a public chatbot.
The Reality: Even though the bank says "No," the guards are using AI anyway because they need help, but they are doing it in the shadows, creating a security blind spot.

2. The "Box of Toys" Trap (License-First Adoption)

The Issue: Banks often buy huge security software packages from vendors. These packages come with a fancy "AI Module" included in the price tag, like a free toy in a cereal box.
The Reality: The bank buys the box because the vendor says, "It has AI!" But once they get it home, they realize they don't know how to play with the toy. They turn it on, see it doesn't fit with their existing security systems, and leave it in the box.
The Result: They are paying for a high-tech robot guard that sits in the corner, turned off, while the human guards still do all the work manually. They bought the license, but they didn't build the capability.

3. The "Blind Spot" (Attacker Perception Gap)

The Issue: The bank's guards are worried about how they use AI. But they aren't worried enough about how the thieves are using it.
The Analogy: Imagine the thieves are using AI to write perfect, undetectable phishing emails (fake messages) that look exactly like they come from your boss. Meanwhile, the bank's guards are still using old-fashioned methods to spot them.
The Gap: The defenders are playing chess, but the attackers are playing 4D chess with AI assistance. The defenders often don't realize the game has changed, so they aren't building defenses against AI-powered attacks.

4. The "Black Box" Fear (Trust & Robustness)

The Issue: If an AI guard says, "Stop that person!" the human guard needs to know why.
The Problem: Current AI often works like a "Black Box." It gives an answer, but it can't explain its reasoning.

The Auditor's Problem: If a regulator (the building inspector) asks, "Why did you arrest that person?" and the answer is "The computer said so," the bank gets in trouble. They need a paper trail.
The Human Problem: If the AI makes a mistake once, the human guard stops trusting it entirely. "I can't explain why it flagged this, so I'm just going to ignore it."
The Fragility: If a thief knows how to trick the AI (like wearing a hat that confuses a facial recognition camera), the whole system fails. The banks are scared to use the AI because they haven't tested if it can handle being tricked.

What the Data Says

The researchers talked to 6 experts and surveyed 14 security professionals. Here is what they found:

Everyone loves the idea: 71% of people think AI will be the main way banks fight crime in 5 years.
But right now, it's rare: Most people only use it occasionally, or not at all, because they don't trust it.
The biggest fear: 57% of people are worried about "adversarial risks"—meaning, they are scared the bad guys will hack the AI itself.

The Solution: Three New Rules

The authors suggest three simple rules to make AI trustworthy in banking:

The "Human-in-the-Loop" Rule: Don't let the AI guard make the final decision alone. If the AI is unsure, it must ask a human. Also, if a guard must use a public AI tool, the bank should provide a safe, monitored version so they don't have to use the "shadow" version.
The "Stress Test" Rule: Before you let the AI guard on duty, you must test it. Try to trick it. Try to confuse it. Make sure it doesn't break when the bad guys try to hack it. And keep a logbook of every time you retrain or update the AI, so auditors can see it's being watched.
The "Turn It Off" Rule: Don't just turn on every AI feature because you paid for it. Only turn on the specific AI tools that fit your specific security workflow. If a feature isn't being used or integrated, keep it disabled so it can't be hacked.

The Bottom Line

AI is a powerful engine, but right now, financial institutions are trying to drive it without a steering wheel, without a map, and without a seatbelt. They are afraid to use it because they don't understand how it works, they can't explain it to the regulators, and they are scared the bad guys will hack it first.

To fix this, banks need to stop buying "AI in a box" and start building "AI with a plan." They need to treat AI not just as a smart tool, but as a security risk that needs its own guard dog.

1. Problem Statement

Financial institutions operate in a high-stakes environment characterized by strict regulatory oversight and evolving cyber threats. While Artificial Intelligence (AI) and Machine Learning (ML) are widely touted as accelerators for Cyber Threat Intelligence (CTI)—promising faster detection, broader coverage, and automated pattern recognition—trustworthy production deployment in the financial sector remains scarce.

The core problem is a "translation gap" between technical research and operational reality:

Research vs. Practice: Existing literature focuses heavily on isolated task performance (e.g., accuracy benchmarks on public datasets) but lacks evidence of how AI integrates into real-world CTI workflows, satisfies regulatory audit requirements, or withstands adversarial manipulation.
The Trust Deficit: Adoption is hindered not just by model accuracy, but by socio-technical barriers including lack of explainability, inability to defend decisions in audits, fragmented data infrastructures, and a shortage of hybrid expertise (cybersecurity + AI).
Security Risks: There is a growing concern that AI models themselves are vulnerable to adversarial attacks (e.g., data poisoning, model evasion) and that their deployment creates new attack surfaces without adequate governance.

2. Methodology

The authors employed a mixed-methods, user-centric research design combining three distinct phases to triangulate findings:

Systematic Literature Review (SLR):
- Scope: 330 publications screened from 2019–2025 across five major databases (IEEE, ACM, Springer, Elsevier, AIS) and top security conferences (NDSS, USENIX).
- Selection: 12 finance-relevant studies were retained for deep analysis.
- Focus: The review assessed the extent to which prior work addressed trustworthy, deployable AI in finance, specifically looking for evidence of human factors, integration, and adversarial robustness.
Semi-Structured Interviews:
- Participants: 6 practitioners from mid-to-large financial institutions (banks) and consulting firms. Roles included CISOs, Heads of Cyber Defense, and AI specialists.
- Process: 45–60 minute sessions exploring practical experiences, barriers, and future expectations.
- Analysis: Thematic analysis using Mayring's content analysis methodology with high inter-coder reliability ( $\kappa = 0.93$ ).
Exploratory Survey:
- Participants: 14 responses from cybersecurity officers, consultants, and IT specialists (including some adjacent sectors like Energy and Education for triangulation).
- Content: Quantitative and qualitative questions regarding AI adoption frequency, perceived barriers (technical, human, regulatory), and experiences with adversarial threats.
- Analysis: Descriptive statistics and frequency analysis, interpreted as indicative signals rather than population estimates due to the small, non-random sample.

3. Key Contributions

The paper makes three primary contributions to the field of AI security in finance:

Domain-Specific Evidence Map: It provides the first systematic synthesis showing that while AI improves accuracy in isolated tasks (fraud/phishing detection), the literature overwhelmingly neglects explainability, human-AI collaboration, and resilience against adversarial manipulation in financial CTI contexts.
Empirical Identification of Socio-Technical Barriers: It moves beyond theoretical models to identify four recurrent "failure modes" observed in practice that prevent trustworthy deployment.
Actionable Operational Safeguards: It proposes three concrete, security-oriented recommendations for practitioners and policymakers to mitigate risks in AI-enabled CTI deployments, supported by released artifacts (search strings, coded matrices, and survey instruments) for replication.

4. Key Results & Findings

A. Adoption Status (RQ1)

Exploratory, Not Operational: AI use in financial CTI is currently limited to "shadow" experiments or vendor-bundled features that remain disabled.
The "License-First" Trap: Institutions often purchase AI modules as part of SIEM/SOAR packages without a clear strategy for integration. Consequently, 57.1% of survey respondents reported daily/weekly use of AI tools, but these are rarely integrated into core CTI workflows.
Future Outlook: 71.4% of respondents expect AI to become central to financial cybersecurity within five years, highlighting a divergence between strategic optimism and current operational reality.

B. Barriers to Adoption (RQ2)

Three primary barriers were identified:

Fragmented Infrastructure: Heterogeneous data sources and legacy systems prevent the creation of comprehensive datasets, leading to noisy, unreliable AI outputs.
Skills Gap: A severe shortage of professionals who possess both cybersecurity operational knowledge and AI/ML expertise.
Regulatory & Audit Constraints: Strict requirements for explainability and auditability mean that "black box" models are unusable. If an AI decision cannot be justified to an auditor, it cannot be used for incident response.

C. Trust and Robustness (RQ3)

Trust is Conditional: Trust depends on interpretability and auditability, not just accuracy. 50% of respondents cited trust in AI recommendations as a "major challenge."
Adversarial Reality: 35.7% of respondents reported direct encounters with adversarial AI threats (e.g., model evasion, data poisoning).
Missing Security for AI: There is a lack of continuous monitoring for model drift, adversarial testing, and audit-ready documentation. Defenders often feel they are "one step behind" attackers who are already leveraging AI.

D. The Four Recurring Failure Modes

The study synthesizes findings into four critical patterns hindering trustworthy AI:

Shadow AI in the Wild: Employees bypass institutional controls to use public LLMs for CTI tasks, creating unvetted data-handling paths.
License-First Enablement: Adoption driven by vendor bundling rather than operational need, leading to unused or poorly integrated features.
Attacker-Perception Gaps: Defenders underestimate the sophistication of AI-driven offensive tactics (e.g., AI-generated phishing), leading to inadequate threat modeling.
Missing Security for the AI: Lack of robustness evaluation, drift monitoring, and audit trails for the AI models themselves.

5. Significance and Recommendations

The paper concludes that trustworthy AI-driven CTI is a security problem as much as a performance problem. To bridge the gap between potential and practice, the authors propose three operational safeguards:

Controlled Analyst-in-the-Loop:
- Treat analyst-AI interaction as a security boundary.
- Mandate auditable rationales for AI outputs (provenance links).
- Default low-confidence outputs to human review.
- Provide approved, monitored internal pathways for AI assistance to prevent "shadow" usage.
Security Assurance Gates & Continuous Monitoring:
- Treat AI components as attackable systems.
- Implement pre-deployment gates: robustness checks (adversarial testing), evaluation on representative data, and documented sign-offs.
- Maintain audit-ready artifacts (model versions, retraining triggers, decision logs) for continuous monitoring and rollback capabilities.
Attack-Surface Reduction via Capability-Driven Integration:
- Move away from "license-first" to "capability-first" enablement.
- Explicitly define where AI influences workflows (e.g., enrichment) and where it is restricted (e.g., autonomous blocking).
- Disable-by-default: Non-integrated AI modules should be disabled to prevent unused but reachable functionality.

Overall Impact: This study provides a critical reality check for the financial sector, demonstrating that without addressing governance, explainability, and the security of the AI models themselves, technical advancements in AI will fail to translate into secure, operational value.