Dual-Node NVIDIA DGX Spark over Tailscale: A Remote-Access Testbed for Distributed LLM Training and Cyber-Threat-Intelligence Fine-Tuning

Technical Summary: Dual-Node NVIDIA DGX Spark over Tailscale

Problem Statement
While compact AI systems have democratized local language model experimentation, there is a lack of practical documentation regarding multi-node distributed training on desktop-class accelerators. Existing literature largely assumes datacenter environments with mature schedulers and high-end interconnects. Furthermore, small laboratories and security-sensitive projects often lack the infrastructure to securely access remote hardware for both research and instruction. This report addresses the gap by investigating whether two compact NVIDIA DGX Spark systems can be configured as a reliable, reproducible distributed training testbed accessible remotely, and whether such infrastructure can simultaneously support domain-specific fine-tuning and educational workflows.

Methodology
The study deployed a two-node proof-of-concept cluster at Grambling State University using the following architecture and procedures:

• Hardware: Two NVIDIA DGX Spark systems, each equipped with a GB10 Grace Blackwell SoC and 128 GB of unified memory.
• Network Topology: The system utilized a dual-plane architecture to separate management traffic from training data:
  • Management Plane: Remote administration (SSH) was conducted over a Tailscale mesh VPN.
  • Training Plane: A dedicated 200 Gb/s QSFP56 direct fiber link connected the nodes via interface enp1s0f1np1 on a private, non-routable subnet (192.168.100.0/24) with Jumbo frames (MTU 9000).
• Software Stack: The deployment used NVIDIA NGC PyTorch containers (version 25.10) with host networking to ensure direct access to the fiber interface. The training framework relied on PyTorch torchrun, Distributed Data Parallel (DDP), and NCCL.
• Configuration: The experiment trained a depth-20 NanoChat model with a 2,048-token context. Each node processed a local batch of 32 sequences, resulting in a global batch of 64 sequences (131,072 tokens per step).
• Operational Adjustments: Critical patches were applied to the upstream NanoChat repository, specifically modifying the step-zero evaluation logic to prevent NCCL watchdog timeouts during initialization. Persistent network configurations (via nmcli/netplan) and container memory limits (memlock, stack) were enforced to ensure stability.
• Downstream Tasks:
  • Cyber-Threat-Intelligence (CTI): A dataset was constructed from 77 CISA advisories mapped to MITRE ATT&CK identifiers, generating 338 training and 37 validation conversations. This was used for Supervised Fine-Tuning (SFT).
  • Education: The cluster supported a 400-level AI course (CS 426) and a CompTIA Security+ POGIL (Process Oriented Guided Inquiry Learning) query engine (CBS 255), utilizing the same nodes for inference via an Ollama-hosted LLM.

Key Results

• Training Performance: The dual-node run sustained a step time of approximately 69.4 seconds, achieving an aggregate throughput of ~1,890 tokens/second. Over four days, the system processed approximately 653 million tokens.
• Scaling Observation: The author notes a 3.5× reduction in wall-clock time compared to a single-node projection (estimated at 14 days vs. 4 days). However, they explicitly state this is not a controlled scaling efficiency claim, as the single-node baseline was estimated rather than measured under matched conditions, and the global batch size doubled with the node count.
• CTI Fine-Tuning Evaluation: An automated evaluation using an LLM-as-a-judge (Ollama-hosted llama3.1:8b) compared a baseline checkpoint against a CTI-augmented checkpoint on 17 held-out prompts.
  • CTI-specific categories (Technique lookup, Name to technique, Threat actor) showed improvements (e.g., +1.33 for "Name to technique").
  • General knowledge categories regressed (e.g., -1.67 for "General concept").
  • The overall score shifted modestly from 2.06 to 2.29 on a 0–10 scale. The author interprets this as evidence of a domain-vocabulary shift rather than a strong factual model.
• Educational Integration: The infrastructure successfully supported concurrent research and teaching workloads, with students accessing the cluster remotely via Tailscale for hands-on inspection of training artifacts and AI-generated practice quizzes.

Significance and Claims
The paper positions itself primarily as a systems proof of concept and a reproducibility reference for small laboratories, rather than a study of novel learning algorithms or definitive scaling efficiency.

• Feasibility: The study establishes that compact, desktop-class AI systems can be linked via dedicated fiber and managed remotely via mesh VPNs to create a functional distributed training environment.
• Operational Lessons: It documents critical engineering details often omitted in high-level reports, including the necessity of separating management and data planes, handling NCCL timeouts, persistent network configuration, and rank monitoring strategies (noting that "rank silence is not rank failure").
• Dual-Use Infrastructure: The work demonstrates that a modest local cluster can serve multiple, otherwise siloed functions: distributed pretraining research, downstream domain adaptation, and certification-aligned instruction.
• Modesty of Claims: The author explicitly avoids overstating performance. They clarify that the single-node baseline was estimated, the CTI evaluation used a single judge model with a small sample size (17 prompts), and the parameter count for the depth-20 model was reported as configured depth rather than independently computed. The report concludes that a matched benchmark is required to convert this proof of concept into a controlled performance study.

In summary, the paper provides a documented, reproducible blueprint for small labs to build secure, remote-accessible distributed LLM training testbeds using consumer-grade high-performance hardware, emphasizing practical deployment over theoretical scaling limits.