💬 NLP

VectraYX-Nano: A 42M-Parameter Spanish Cybersecurity Language Model with Curriculum Learning and Native Tool Use

The paper introduces VectraYX-Nano, a 42M-parameter Spanish language model trained from scratch on a curated cybersecurity corpus with curriculum learning and native Model Context Protocol (MCP) tool integration, demonstrating effective performance on commodity hardware and providing new insights into the relationship between corpus density and tool-use capabilities at the nano-scale.

Original authors: Juan S. Santillana

Published 2026-05-15

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Juan S. Santillana

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: A Tiny, Specialized Spanish Security Assistant

Imagine you have a security guard for your computer network. Usually, these guards are huge, expensive, and only speak English. But what if you needed a guard for a specific neighborhood in Latin America who speaks Spanish, understands local slang, and can work on a tiny, cheap device (like a Raspberry Pi) without needing the internet?

That is VectraYX-Nano. It is a very small Artificial Intelligence (AI) model (only 42 million "brain cells," compared to the billions in famous models) designed specifically to help cybersecurity analysts in Spanish-speaking regions. It is unique because it was built from scratch to speak Spanish, understand cyber threats, and know how to use tools to get real-time answers.

1. The Brain Food: How They Taught It (The Corpus)

To teach an AI, you have to feed it data. The researchers didn't just dump random internet text on it. They cooked a specific "three-course meal" using a Curriculum Learning approach:

Course 1: The Conversation Starter. First, they fed the model 42 million tokens of Spanish subtitles and chat logs.
- Analogy: Think of this as teaching a child how to say "Hello," "How are you?" and "Goodbye" before teaching them math. If you skip this, the AI might know everything about viruses but answer "Hello" with a technical report on a virus instead of a greeting.
Course 2: The Technical Deep Dive. Next, they fed it 118 million tokens of cybersecurity data (vulnerability reports, Wikipedia articles on hacking, security blogs).
- Analogy: Now the child knows how to talk, so you teach them the subject matter: "This is what a virus is," "This is how hackers break in."
Course 3: The Tool Kit. Finally, they gave it 10 million tokens about how to use security tools (like looking up threats in a database).
- Analogy: The child now learns how to use a dictionary or a map. Instead of memorizing every single fact, they learn how to ask for the answer.

The "Replay" Trick: A common problem in teaching AI is "catastrophic forgetting"—when you teach them new stuff, they forget the old stuff. To fix this, the researchers used a Replay Buffer.

Analogy: Imagine you are teaching a student history. When you start teaching them about the 20th century, you don't just stop talking about the 19th century. You keep reviewing a little bit of the old lessons (25% of the time) while learning the new ones. This kept the AI from forgetting how to speak Spanish while it learned about cyberattacks.

2. The "Register" Surprise (The Bootstrap Lesson)

One of the paper's most interesting findings is about what you teach first.

The researchers tried two different "Course 1" options:

Option A: Subtitles and chat logs (OpenSubtitles).
Option B: Encyclopedic web articles (mC4-ES).

Surprisingly, the model trained on Option B (encyclopedic articles) actually got better math scores (lower "loss" numbers) during training. However, when they asked it to chat, it sounded like a boring encyclopedia. It would answer "Hello" with a paragraph about travel history.

The model trained on Option A (subtitles) had slightly worse math scores but sounded like a real person.

The Lesson: For a tiny AI, the first thing it learns sets its personality. If you teach it to sound like a textbook first, it will stay like a textbook forever, even if you try to teach it to chat later. You have to teach it to sound like a human first.

3. The Magic Tool: MCP (Model Context Protocol)

Cybersecurity changes every day. New viruses appear daily. An AI cannot memorize every single new virus.

The Problem: If you ask a standard AI, "Is this new virus dangerous?", it might guess or make things up (hallucinate) because it doesn't know about today's news.
The Solution: VectraYX-Nano is trained to use tools. It doesn't try to memorize the answer. Instead, it learns to say: "I don't know, but let me check the database."
How it works: The AI is connected to a system called MCP. When it needs info, it sends a structured request (like a JSON command) to a server that has the latest data, gets the answer, and reads it back to you.
The Discovery: The researchers found that for the AI to learn this "ask for help" behavior, the training data needed to be dense. If the training data was 99% normal chat and only 1% tool-use, the AI was too shy to use the tool. But if they increased the tool-use data to about 1 out of every 20 examples, the AI suddenly started using the tools reliably.

4. Size Matters (But Not How You Think)

The model is tiny (42 million parameters).

The Good News: It runs on a Raspberry Pi (a $35 computer) in under a second. It fits in your pocket.
The Limitation: Because it is so small, it can't be a "genius" at everything. It can't reason through complex, multi-step logic as well as a giant model.
The Trade-off: It trades "memorizing facts" for "knowing how to look them up." It's better to have a small assistant that knows how to check the facts than a giant assistant that confidently lies about them.

5. Safety and "Red Teaming"

Since this model is trained on hacking data (to understand how attacks work), the researchers worried it might become a "bad actor."

The Test: They tried to trick the AI into doing dangerous things (like deleting files or stealing data).
The Result: The AI didn't actually do the dangerous things. It mostly gave vague answers or refused. However, it didn't have a built-in "moral compass" to say "No" explicitly.
The Fix: The paper suggests that safety shouldn't rely on the AI's brain alone. Instead, the system running the AI (the runtime) should act as the bouncer, blocking any dangerous commands before they are executed.

Summary of Key Takeaways

Teach the personality first: For small AI models, the first data you feed them determines how they sound. If you want a chatbot, start with chat data, not encyclopedias.
Don't forget the past: When teaching new topics, keep reviewing the old ones (Replay Buffer) so the AI doesn't forget its language skills.
Tools over Memory: For small models, it's better to teach them how to use a tool to find answers than to force them to memorize everything.
Density is key: To teach an AI to use tools, you need a lot of examples of tool-use in the training data. If it's too rare, the AI won't learn to do it.
Edge Deployment: This proves you can build a useful, specialized AI that runs on cheap, local hardware without needing the cloud, which is great for security teams who can't send sensitive data to the internet.

What they released: The researchers didn't just write a paper; they released the "recipe" (code), the "ingredients" (data construction guide), and the final "dish" (the AI model weights) so anyone can build their own version.

Technical Summary: VectraYX-Nano

Problem Statement

The paper addresses two intersecting gaps in the Large Language Model (LLM) ecosystem relevant to security operations centers (SOCs) in Latin America:

Language Gap: The most capable open-weight chat models are predominantly trained on English corpora, leaving Spanish-speaking analysts with limited options. Existing Spanish models often lack technical accuracy or require closed-source, non-auditable APIs.
Domain Gap: Cybersecurity-specialized models exist but are almost exclusively trained on English data, failing to capture Latin American regional terminology (e.g., specific CSIRT vocabularies like CCN-CERT, INCIBE) and threat contexts.
Deployment Constraints: Security teams often operate in air-gapped or on-premise environments where data cannot leave the network, necessitating small, edge-deployable models that can be audited and retrained locally.
Tool Integration: While tool-augmented LLMs are emerging, existing solutions often rely on frontier models behind agentic loops. There is a lack of small, native tool-using models that can reliably invoke external systems (via the Model Context Protocol, or MCP) to retrieve real-time threat intelligence (CVEs, IOCs) rather than hallucinating from a static training cutoff.

Methodology

1. Corpus Construction (VectraYX-Sec-ES)

The authors assembled a 170M-token Spanish cybersecurity corpus using a distributed pipeline of eight virtual machines at a cost of approximately $25 USD. The corpus is partitioned into three phases aligned with a curriculum learning strategy:

Phase 1 (Conversational): 42M tokens derived from OpenSubtitles-ES, OASST1, and custom curated dialogues to establish a chat register.
Phase 2 (Cybersecurity): 118M tokens including NVD CVE entries, a filtered Spanish Wikipedia subset (82M tokens, the largest component), and translated security blogs.
Phase 3 (Tooling): 10M tokens from ExploitDB, HackTricks, and OWASP.
SFT Corpus: A separate 28M-token dataset containing 13K conversations, 4,030 CVE Q&A pairs, and 6,327 tool-use traces grounded in a real on-premise CVE/IOC database.

2. Architecture

VectraYX-Nano is a 41.95M-parameter decoder-only Transformer trained from scratch. Key architectural choices include:

Modern Small-LLM Stack: RMSNorm, SwiGLU, RoPE, and weight-tied embeddings.
Stability Mechanisms: Grouped-Query Attention (GQA) with 8 query heads and 2 KV heads (4:1 ratio) to reduce KV cache and parameters; QK-Norm to stabilize training on small batches; and a $z$ -loss auxiliary to prevent logit drift.
Tokenizer: A domain-balanced 16,384-token byte-fallback BPE tokenizer trained on a 50/50 mix of conversational and technical text. It includes 27 reserved special tokens for chat roles, tool calls, and domain-specific concepts (e.g., <|cve|>, <|tool_call|>).

3. Training Strategy

Curriculum Learning with Replay: The model undergoes three-phase continual pre-training. Crucially, explicit replay buffers (25% in Phase 2, 10% in Phase 3) are used to mitigate catastrophic forgetting of the conversational register established in Phase 1.
Supervised Fine-Tuning (SFT): A mini-curriculum approach is used during SFT, front-loading conversational data before introducing tool-use traces to prevent the model from overfitting to JSON formatting at the expense of natural language.
Tool Use: The model is trained to emit syntactically valid <|tool_call|> JSON segments that are executed by an external MCP runtime, not the model itself.

Key Contributions

VectraYX-Nano Model: The first published Spanish-native, decoder-only cybersecurity LLM trained from scratch with native MCP tool-use support, optimized for edge deployment (81 MB in F16, ~20 MB in 4-bit).
Corpus and Pipeline: A low-cost ($25) distributed pipeline for constructing a 170M-token Spanish cybersecurity corpus, including a unique "Pikachu" in-house mirror of translated NVD CVEs and regional threat intelligence.
Curriculum with Replay: Demonstration that a three-phase curriculum (Conversational → Cybersecurity → Tooling) with replay buffers prevents catastrophic forgetting and yields monotonic loss descent.
Empirical Findings on Register and Density:
- Loss-vs-Register Inversion: The paper reports that using a lower-perplexity bootstrap corpus (mC4-ES) results in worse conversational behavior compared to a higher-perplexity corpus (OpenSubtitles-ES). At the 42M scale, the bootstrap corpus's "register" (style) dominates downstream chat quality, a factor that SFT alone cannot fully overwrite.
- Tool-Use Density Threshold: The authors identify that the failure of small models to emit tool calls (B4 score of 0.000) is not a hard capacity limit but a corpus-density artifact. Increasing the ratio of tool-use examples to prose from 1:211 to 1:21 significantly improves tool-selection accuracy (B4) on both 42M and 260M models, shifting the first-token prior from prose to <|tool_call|>.

Results

Performance: The released model achieves a conversational gate score of 0.78 ± 0.05 (human-graded) on a held-out benchmark.
Efficiency: It runs at sub-second time-to-first-token on commodity hardware (Raspberry Pi 4) and achieves 6–10 tokens/second on the Pi 4, compared to 1–2 tokens/second for a 0.5B baseline.
Ablation Studies:
- Models trained with OpenSubtitles-ES bootstraps outperform those trained with mC4-ES on conversational metrics, despite the latter having lower pre-training loss.
- A 260M parameter "mid-tier" model trained from scratch with the same curriculum shows improved B1 (keyword recall) and B3 (command completion) but still hits a 0.000 floor on tool selection (B4) unless the tool-use corpus density is increased.
Safety: Automated red-team evaluations on 499 adversarial prompts show the model does not autonomously emit dangerous tool calls. However, it lacks explicit refusal training, relying on runtime-level filtering for safety.

Significance and Claims

The paper positions VectraYX-Nano as a proof-of-concept for edge-deployable, auditable AI assistants for Latin American security teams. Its primary significance lies in demonstrating that:

Nano-scale models can be effective for specific domains if the training data is carefully curated and the curriculum is designed to establish the correct stylistic register before domain specialization.
Tool use in small models is feasible but is gated by the density of tool-use examples in the training data, not just parameter count.
Perplexity is not the sole objective for chat models; at small scales, matching the response register of the bootstrap corpus is more critical for user-facing behavior than minimizing perplexity.

The authors release the corpus construction recipes, training scripts, configurations, GGUF weights, and a benchmark suite (VectraYX-Bench) to enable reproducibility and further research in Spanish cybersecurity NLP. They explicitly note that the model is not intended to compete with frontier 70B+ models on open-domain reasoning but serves a specific niche in on-premise, tool-augmented threat analysis.