Technical Summary: NeuraDock Visual Cognitive Load Agent

Problem Statement

The increasing integration of Large Language Models (LLMs) into scientific software offers accessibility but introduces a critical "category error" in domains like electroencephalography (EEG). While LLMs possess broad knowledge of EEG terminology and general processing, they lack specific awareness of a particular device's physical constraints, the current software's implemented algorithms, the precise meaning of computed result fields, and the scientific limits of specific inferences.

In low-channel EEG, where spatial coverage is sparse and signal quality varies, this lack of boundary awareness leads to plausible but unsupported interpretations. A fluent model might recommend frontal alpha asymmetry for a posterior-only montage, describe a workflow that does not exist, or translate an engineering quality flag into a neurological diagnosis. Furthermore, placing stochastic code generation within the numerical analysis path introduces liability and variability, while sending raw EEG data to external models raises privacy concerns regarding sensitive neural data.

Methodology

The authors propose NeuraDock Agent, an open-source architecture that strictly separates a deterministic local EEG engine from a hardware-aware language layer.

System Architecture

1. Local Deterministic Core: A versioned Python engine handles all numerical computation. It parses recordings, performs quality control (QC), executes reviewed spectral workflows, and writes machine-readable artifacts (JSON results, Markdown reports, figures). Raw EEG data and dense per-sample arrays remain local and are never sent to the LLM.
2. Allowlisted Interpretation Payload: The LLM receives only a compact, allowlisted summary of the results. This payload explicitly excludes raw samples, full PSD vectors, and local file paths. It includes only fields necessary for explaining quality, parameters, aggregate trends, and stated limits.
3. Versioned Context Layer: The LLM operates within a constrained context pack containing:
   • Hardware Profile: Specifics of the 7-channel dry-electrode system (CP5, CP6, PO3, PO4, O1, Oz, O2 at 250 Hz) and engineering QC thresholds.
   • Implementation Registry: A closed list of reviewed workflows (e.g., Signal Quality, PSD, Visual Cognitive Load, Rest/Task Comparison).
   • Result Schema: Definitions of output fields and their scientific limits.
   • Scientific Boundaries: Explicit statements on what inferences are not justified (e.g., no causal claims, no absolute cognitive load indices).

The Visual Cognitive Load Workflow

As a case study, the system implements a "Visual Cognitive Load" workflow. This is a relative, within-recording heuristic based on posterior alpha suppression (8–13 Hz). It calculates a composite score from standardized log-alpha power, peak frequency, and right-minus-left asymmetry. Crucially, the system treats this as a relative ranking (low/medium/high) within a session, not an absolute diagnostic index.

Key Contributions

1. Deterministic, Privacy-Bounded Architecture: A system design where the LLM acts solely as a constrained interpretation and planning layer, while the local engine remains the source of numerical truth.
2. Documented Workflow Limits: A formal definition of the Visual Cognitive Load workflow, explicitly stating its quality and interpretive boundaries (e.g., it is not a diagnosis of attention or fatigue).
3. System-Level Evidence: Empirical validation of numerical repeatability, request minimization (payload size), failure isolation (local artifacts survive LLM failures), and deterministic responses to controlled signal perturbations.
4. Boundary-Awareness Benchmark: A novel evaluation framework using 36 cases across four context conditions (Generic, Hardware, Hardware+Implementation, Full Context) and two LLMs to measure exact recognition of physical, implementation, result, and scientific boundaries.

Results

Repeatability and Safety

• Determinism: 12 recordings produced identical structured results over ten repetitions. End-to-end runs (results, reports, figures) yielded identical hashes across three repetitions.
• Data Minimization: The LLM request payload was significantly smaller than the raw recording (approx. 23 KB vs. 8 MB), with explicit confirmation that raw EEG and dense arrays were absent.
• Failure Isolation: Local artifacts (results, reports, traces) were preserved even when the LLM service failed (HTTP 400, malformed output, connection refusal).

Boundary-Awareness Benchmark

The benchmark evaluated 288 outputs (36 cases × 4 contexts × 2 models).

• Decision Accuracy: Exact four-way boundary classification (Supported, Conditional, Unsupported, Not Implemented) improved from 58.3% (Generic prompt) to 79.2% (Full Context).
• Safe Responses: The rate of fully correct, fact-complete, and false-claim-free responses increased from 26.4% to 66.7%.
• Rejection Calibration: The generic model was overly conservative, rejecting 27.8% of feasible requests. Full context reduced this "over-refusal" rate to 8.3% while maintaining a low unsupported acceptance rate (1.4%).
• Context Ablation: Adding hardware context provided the largest initial gain. However, "Full Context" was not uniformly optimal; a "Hardware + Implementation" subset sometimes achieved a slightly higher strict safe-response rate, suggesting that indiscriminate context expansion can introduce noise or over-refusal.

Limitations Observed

• Flatline Detection: The current QC workflow failed to detect 15-second flatline dropouts, a concrete negative result.
• Error Types: Residual errors in the full-context condition were primarily due to difficulty preserving fine-grained conditional semantics (e.g., distinguishing between "physically plausible but unimplemented" vs. "unsupported") and omitting required facts in complex orchestration scenarios.
• Physiological Validation: The exploratory Rest/Task pilot showed mixed results (non-significant p-values), reinforcing that the system does not yet establish a validated absolute cognitive-load index.

Significance and Claims

The paper positions NeuraDock Agent not as a clinical diagnostic tool or a replacement for expert EEG analysis, but as a systems and evaluation framework.

• Calibration over Refusal: The primary contribution is demonstrating that "boundary awareness" allows an agent to calibrate when to accept, qualify, or refuse a request, rather than simply refusing everything to avoid hallucination.
• Grounding Mechanism: The results support hardware- and implementation-aware grounding as a practical mechanism for low-channel EEG agents. It ensures that the LLM's behavior is constrained by the physical device contract and the current software capabilities.
• Modest Scope: The authors explicitly state that the software does not diagnose cognition, replace expert analysis, or establish a clinically validated cognitive-load index. The system is intended for research, engineering, education, and prototyping.
• Future Direction: The work motivates a shift from static, massive context prompts to selective retrieval of relevant context modules (hardware, implementation, scientific limits) to optimize decision-making without overwhelming the model.

In summary, the paper argues that for scientific agents to be trustworthy, they must operate inside explicit, versioned, and testable boundaries, separating the stochastic nature of language generation from the deterministic requirements of scientific computation.