Agentic coding without the cloud: evaluating open-weight large language models on longitudinal data preparation tasks

Technical Summary: Agentic Coding without the Cloud

Problem Statement
Quantitative health and social sciences face a persistent bottleneck in data preparation, a labor-intensive process involving variable recoding, index construction, and multi-wave harmonization. While Large Language Models (LLMs) and agentic frameworks offer potential automation, their adoption in research involving sensitive personal data is constrained by governance requirements. Trusted Research Environments (TREs) and data access agreements typically prohibit transmitting individual-level data to external cloud-based LLMs. Although locally deployable open-weight models offer a privacy-preserving alternative, existing benchmarks (e.g., HumanEval, SWE-bench) focus on general software engineering or Python, failing to capture the domain-specific reasoning required for longitudinal survey data preparation in R, where tasks involve interpreting imperfect metadata and reconciling heterogeneous coding systems.

Methodology
The authors developed an open-source framework, RRBench, to evaluate the efficacy of open-weight LLMs on realistic data preparation tasks within a local, air-gapped environment.

• Ground-Truth Dataset: The benchmark utilizes the Next Steps Core dataset, a cleaned, research-ready version of a British longitudinal cohort study (ages 14–32). The ground truth consists of 20 distinct tasks derived from experienced cohort researchers, involving the creation of 102 variables across demographic, education, health, and socioeconomic domains.
• Agentic Workflow: The system employs SmolAgents, a lightweight framework that does not require native tool-calling capabilities, allowing the use of smaller models on consumer-grade hardware. The agent receives input data files, metadata (in JSON format containing variable names, labels, and value labels), and a task prompt. It iteratively generates, executes, and debugs R scripts within a closed local environment.
• Model Selection: Eight open-weight models were evaluated, ranging from 8B to 35B parameters, including both dense and Mixture-of-Experts (MoE) architectures (e.g., Gemma 4, Qwen3.5/3.6, Ministral). Models were selected based on their ability to handle context windows of at least 100,000 tokens and their recent release (within 12 months of May 2026).
• Evaluation Metrics: Performance was assessed by comparing the LLM-generated datasets against the human-curated ground truth using:
  • Completeness: The proportion of expected variables present in the output.
  • Correctness: The proportion of recovered variables meeting strict agreement thresholds (≥95% cell-level agreement for categorical; normalized RMSE ≤10⁻⁴ for continuous).
  • Balanced Performance: The product of completeness and correctness.
  • Downstream Impact: Regression analyses were conducted to measure how data errors propagated into substantive findings (e.g., regression coefficients for BMI and income).

Key Results

• Model Performance: State-of-the-art models in the 31–35B parameter range (specifically Gemma 4 31B and Qwen3.6-35B-A3B) achieved the highest performance. Gemma 4 31B completed 75% of tasks fully in a single pass, with a balanced variable performance of 85.9% and an average task completion rate of 87.9%.
• Completeness vs. Correctness: While most models achieved high completeness (generally >95%), correctness was the primary differentiator. Smaller models (e.g., Ministral-3-14B) frequently produced plausible-looking but incorrect preprocessing steps, such as mislabeling missing values or applying incorrect recoding logic.
• Task Complexity: Performance was strongly negatively correlated with task complexity. Tasks involving universal definitions (e.g., sex, BMI) were reliably completed even by smaller models. Conversely, tasks requiring the interpretation of survey-specific metadata and complex derivation logic (e.g., housing tenure, household income) proved difficult; housing tenure was not fully completed by any model in any run.
• Downstream Consequences: Datasets generated by the best-performing model (Gemma 4 31B) yielded regression coefficients identical to the ground truth. In contrast, datasets from weaker models contained "silent errors" (e.g., incorrect NA handling) that materially altered regression coefficients, such as reversing the direction of an economic activity coefficient.
• Prompt Sensitivity: Performance dropped significantly when using "lite" prompts with fewer explicit instructions, highlighting the models' reliance on detailed guidance for complex derivation logic.

Significance and Claims
The paper claims to provide the first benchmark specifically designed for evaluating locally deployable LLMs on longitudinal survey data preparation in R. Its primary contribution is demonstrating that open-weight models running on consumer-grade hardware can achieve high fidelity in data preparation tasks without transmitting sensitive data to the cloud, offering a viable path for AI-assisted research under strict governance frameworks.

The authors emphasize that while the most performant models are currently restricted to high-end consumer hardware (e.g., single 24GB GPUs), the framework proves that local deployment is feasible. They argue that optimizing metadata to aid LLM understanding may yield greater performance gains than model-side improvements alone. The paper concludes that while these models show promise for "first-pass" script generation, human-in-the-loop review remains essential, particularly for smaller models prone to silent errors. The framework is released as open-source to facilitate community adaptation and expansion to other datasets and programming languages.