Distilling Answer Set Programming Theories from Large Language Models

Technical Summary: Distilling Answer Set Programming Theories from Large Language Models

Problem Statement
Writing Answer Set Programming (ASP) theories from scratch is a labor-intensive task requiring hundreds of rules to cover the operators of non-trivial reasoning tasks. Handwritten theories are often domain-specific and do not transfer across datasets with differing schemas. This paper investigates whether a Large Language Model (LLM), acting as an autonomous agent with a solver in the loop, can distill a complete and correct ASP theory from scratch. The goal is to determine if an LLM can iteratively author a theory given only an empty file, a fixed prompt, and access to training examples, without human-supplied templates or partial rule patterns.

Methodology
The authors propose a dataset-agnostic distillation protocol instantiated on the Visual Question Answering (VQA) domain. The system operates within a "neurosymbolic" framework where the LLM acts as the reasoning module author, while a standard ASP solver (clingo) provides the execution engine.

• The Agent Harness: The LLM is embedded in a sandboxed environment (Docker container) using the OpenCode agent CLI. The model has access to file manipulation tools (read, edit, write, glob, grep) and a restricted bash interface to run uv run solve (to execute the ASP solver on training data) and uv run lint (to check syntax).
• The Protocol:
  1. Initialization: The agent starts with an empty theory file and a minimal system prompt instructing it to produce a theory that derives ans(A) for training examples.
  2. Iteration: The agent reads a few training examples, writes an initial syntactically valid theory, and tests it against the solver.
  3. Feedback Loop: Based on the solver's output (or lack thereof), the agent identifies failure modes, edits the theory, and iterates.
  4. Termination: The loop continues until the agent self-declares completion or a 1-hour time limit is reached.
  5. Evaluation: The final theory is scored on a held-out validation set. The unit of work is the entire theory, authored across many edit cycles.
• Experimental Setup: The study evaluates nine models across three VQA benchmarks:
  • CLEVR: Synthetic 3D scenes focusing on compositional reasoning.
  • GQA: Real-world images with a large vocabulary of objects and relations.
  • CLEVRER: Short videos requiring temporal, causal, and counterfactual reasoning.
  • Model Tiers: Four frontier models (Claude Sonnet 4.6, Claude Opus 4.7, GPT-5, DeepSeek V4 Pro), two mid-tier models, and three smaller open-weights models.
  • Reference Ablation: Experiments vary the inclusion of handwritten reference theories from other datasets (0, 1, or 2 references) to test transferability.

Key Contributions

1. A Unified Distillation Protocol: The authors establish a fixed agent harness, system prompt, and tool set that is identical across all datasets and models. This isolates model capability as the primary variable, removing prompt engineering as a confounding factor.
2. Demonstration of Theory Distillation: The study provides empirical evidence that agents can distill complete ASP theories from scratch. Three of the four frontier models achieved performance matching or exceeding handwritten reference theories on specific benchmarks.
3. Failure Mode Taxonomy: The paper categorizes observed failures into parse errors, grounding failures, "no answer" (theory is silent), and semantic errors. It also analyzes how model scale and reference theories interact with these failure modes.

Results

• Frontier Model Performance:
  • CLEVR: Three frontier models (Sonnet, Opus, DeepSeek V4 Pro) reached 100% accuracy. GPT-5 reached 98.7%.
  • GQA: Sonnet, Opus, and DeepSeek V4 Pro scored between 92.8% and 98.8%, exceeding the handwritten reference ceiling (77.5%). GPT-5 dropped significantly to 41.8%, primarily due to "no answer" failures (the theory failed to derive an answer for most examples).
  • CLEVRER: The three successful frontier models scored between 92.7% and 95.3%. GPT-5 scored 86.7%.
• Impact of Reference Theories:
  • For Sonnet, Opus, and DeepSeek V4 Pro, adding reference theories from other datasets had a negligible effect (within ±3.4 percentage points).
  • For GPT-5, adding references caused a consistent regression in accuracy (drops of 3–19 percentage points). The authors observe that GPT-5 writes significantly shorter theories when given references, suggesting the reference consumes context budget needed for authoring.
• Model Scale and Sub-Frontier Performance:
  • Models below the 27B parameter threshold (e.g., qwen3.5-9b, gpt-oss-20b) largely failed to produce parseable theories or stalled before authoring.
  • DeepSeek V4 Flash (mid-tier) performed well on CLEVR and GQA but degraded on CLEVRER, especially with references.
  • Smaller models exhibited high variance and specific failure modes like malformed tool calls or inability to generate valid clingo syntax.
• Theory Characteristics: Successful frontier models produced theories of varying lengths (e.g., GPT-5 wrote 65 rules for GQA vs. 343 for Opus), yet both achieved high accuracy, indicating that rule count is a function of authoring style rather than coverage.

Significance and Claims
The paper claims that neurosymbolic agents can successfully distill complete, correct ASP theories from scratch, challenging the notion that such theories must be hand-crafted. The study highlights that while frontier models can achieve state-of-the-art results on compositional and real-world VQA tasks, performance is not uniform across all top-tier models; GPT-5's specific failure on GQA suggests that agent loops can be "net-negative" for certain models on specific tasks.

The authors modestly conclude that while the protocol works for high-capability models, the "capability threshold" is not purely a function of parameter scale but involves specific competencies in tool use and syntax generation. They note that the distilled theories are currently static and propose future work to integrate them into training loops (e.g., NS-CL) and to auto-generate the question parsers, rather than relying on existing functional-program annotations. The work does not claim to solve VQA end-to-end but rather demonstrates the feasibility of automating the symbolic reasoning component of neurosymbolic pipelines.