💻 computer science

ChaosBench-Logic v2: Evaluating LLM Logical Reasoning over Dynamical Systems at Scale

The paper introduces ChaosBench-Logic v2, a large-scale benchmark and the CARE evaluation protocol designed to reveal critical logical reasoning failures in LLMs regarding dynamical systems, finding that while models perform moderately on formal deduction, they struggle significantly with regime transitions and exhibit systematic anti-correlation on bifurcation questions.

Original authors: Noel Thomas

Published 2026-05-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Noel Thomas

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are testing a group of students to see how well they understand the rules of a complex board game. Some students are great at memorizing the rulebook and applying it when the rules are written right in front of them. Others, however, seem to get confused when the game gets tricky, or when the question is phrased slightly differently.

This paper, ChaosBench-Logic v2, is a massive new "exam" designed to test Large Language Models (LLMs)—the AI brains behind chatbots—on their ability to reason about dynamical systems. Think of these systems as complex machines (like weather patterns, swinging pendulums, or population growth) that follow strict mathematical laws but can behave in wild, unpredictable ways.

Here is a breakdown of what the researchers did and found, using simple analogies:

1. The Exam: A Giant Leap in Difficulty

The authors created a new benchmark called ChaosBench-Logic v2.

The Scale: The first version had about 600 questions. This new version has 40,886 questions. It's like upgrading from a pop quiz to a final exam that covers an entire library of physics and math problems.
The Content: It tests 165 different "machines" (dynamical systems) using 27 specific logical rules (predicates) and 78 connecting rules (axioms).
The Goal: To see if AI can truly reason about how these systems change over time, or if they are just guessing based on patterns they've seen before.

2. The New Grading System: "CARE"

The researchers realized that a simple "percentage score" (like getting 60% right) is a trap.

The Trap: Imagine a student who is too scared to guess "Yes." They just answer "No" to everything. If 80% of the questions on the test are actually "No," this student gets an 80% score! But they didn't actually learn anything; they just guessed the most common answer.
The Solution (CARE): The authors introduced a new grading protocol called CARE. Instead of just looking at the score, it checks:
- Did they cheat by guessing? (Prior Collapse)
- Are they consistent? If you ask the same question in different words, do they give the same answer?
- Are they balanced? Do they know when to say "Yes" and when to say "No"?

3. The Results: The "Rule-Following" vs. "Intuition" Gap

When they tested 14 different AI models (both paid "proprietary" ones and free "open-source" ones), they found some surprising things:

The "Rule-Following" Superstars: When the exam gave the AI the facts and asked it to use logic to find the answer (like a math word problem), the top models did quite well. They could follow the rulebook perfectly.
The "Intuition" Failures: When the exam asked the AI to predict when a system would change its behavior (like "At what speed does this car start to skid?"), the models performed near-randomly. It was as if they were flipping a coin.
- The Analogy: The AI is great at saying, "If A implies B, and B implies C, then A implies C." But it is terrible at knowing, "If I push this swing hard enough, it will break." It lacks the "physical intuition" or numerical grounding to know the specific tipping points.

4. The "Proprietary" vs. "Open Source" Showdown

Usually, people assume the expensive, closed-source models (like the big tech company models) are strictly better than the open-source ones.

The Twist: The gap isn't uniform.
- The Proprietary models were better at connecting different clues and staying consistent when the question was rephrased.
- However, the Open-Source model (Qwen 2.5-32B) actually crushed the proprietary models on one specific task: interpreting numerical indicators (like reading a speedometer). It was the only model that could reliably interpret the "chaos indicators."

5. The "Anti-Correlation" Problem

The most alarming finding was that two models didn't just get the hard questions wrong; they got them systematically wrong.

The Analogy: Imagine a student who, instead of guessing randomly, has learned a rule that is the exact opposite of reality. If the answer is "Yes," they confidently say "No." If the answer is "No," they say "Yes."
The paper found that for questions about "bifurcations" (tipping points), some models had a negative score, meaning their logic was perfectly inverted. They were confidently wrong.

6. The "Repair" Experiment

The researchers tried to "fix" the AI's answers using a logic solver (a tool that checks if the answers make sense together).

What worked: For simple questions, the tool could fix the AI's mistakes by forcing it to follow the rules.
What failed: For complex, multi-step reasoning questions, "fixing" the answers actually made the AI worse.
The Lesson: This proves there are two types of errors:
1. Consistency errors: "I said A, but then I said not-A." (Fixable by a logic tool).
2. Reasoning errors: "I don't understand the physics of the situation." (Cannot be fixed by a logic tool; the AI needs to actually learn the concept).

Summary

The paper concludes that while AI models are getting better at following logical rules, they still struggle to understand how real-world systems actually behave when numbers and parameters change. They are like students who can memorize the dictionary but can't write a story. The "gap" between rule-following and true understanding doesn't disappear just by making the AI bigger; it requires a different kind of learning.

Technical Summary: ChaosBench-Logic v2

Problem Statement

Standard accuracy metrics on binary reasoning benchmarks often obscure critical failure modes in Large Language Models (LLMs), such as prior collapse (exploiting class priors), inconsistency under paraphrase, and an inability to reason about parameter-dependent dynamics. While LLMs demonstrate strong performance on mathematical and propositional logic tasks, their capacity for logically consistent reasoning over scientific domains—specifically dynamical systems—remains poorly understood. Dynamical systems present a unique challenge: they are deterministic yet non-random, exhibit sensitive dependence on initial conditions, and require the maintenance of formal distinctions (e.g., positive Lyapunov exponents) across multi-step inferences. Existing benchmarks either lack domain-specific First-Order Logic (FOL) ontologies, rely on synthetic ground truth, or fail to scale to real scientific systems.

Methodology

1. Benchmark Design: ChaosBench-Logic v2

The authors introduce ChaosBench-Logic v2, a 66× scale-up of the previous v1 benchmark. The dataset comprises 40,886 questions across 165 dynamical systems (30 manually curated, 135 from the dysts library).

Ontology: The benchmark is grounded in a FOL ontology with 27 unary predicates (e.g., Chaotic, Deterministic, PosLyap, Bounded) organized into three tiers: core dynamical regimes, topological properties, and structural characteristics.
Axioms: These predicates are connected by 78 directed axiom edges (31 implications, 47 exclusions), enabling reasoning chains of up to 5–6 hops. For instance, Chaotic(s) entails Deterministic(s), PosLyap(s), and Sensitive(s), while excluding Random(s) and Periodic(s).
Task Families: The dataset includes 11 task families categorized by difficulty and reasoning type:
- Hard: Regime transition (requires specific bifurcation thresholds, e.g., $r \approx 3.57$ for the logistic map), Cross indicator (multi-indicator reasoning), and Consistency paraphrase.
- Medium: Multi-hop inference, Adversarial (near-miss and misleading cues), and Atomic (single-predicate queries).
- Easy: FOL inference (deduction from explicit premises), Indicator diagnostic (interpreting numerical chaos indicators), and Extended systems (factual recall).

2. Evaluation Protocol: CARE

To address the limitations of standard accuracy, the authors propose CARE (Calibration- and Adversarial-Robust Evaluation). This protocol reports five diagnostics:

Matthews Correlation Coefficient (MCC): The primary metric, chosen for its invariance to class balance and ability to penalize prior collapse.
Macro-family MCC: The mean MCC across task families to prevent dominant, easier families from masking failures on hard tasks.
Calibration: Comparison of predicted TRUE rates against ground-truth TRUE rates to detect prior collapse.
Consistency: MCC measured on paraphrase and perturbation families to assess robustness to surface-level variations.
Coverage: The rate of invalid (unparseable) responses.

Key Contributions

Scalable Benchmark: A 40,886-question dataset covering 165 dynamical systems with a rigorous FOL ontology and ground-truth labels derived from physical properties rather than synthetic constructions.
CARE Protocol: A diagnostic framework that exposes hidden pathologies (prior collapse, inconsistency) often missed by accuracy metrics.
Diagnostic Findings: Identification of a "knowledge-type boundary" between rule-following (FOL deduction) and parameter-dependent reasoning (bifurcation thresholds), along with a per-family decomposition of the proprietary vs. open-source performance gap.

Experimental Results

The authors evaluated 14 models (7 proprietary, 7 open-source).

Overall Performance: Claude Sonnet 4.6 leads with an MCC of 0.601. The proprietary–open-source gap is 0.12 MCC, but Qwen 2.5-32B (0.478) outperforms several proprietary models (GPT-4o: 0.450, Gemini 2.5 Flash: 0.458).
Task Family Hardness:
- Regime Transition: Performance is near-random for all models (MCC = 0.05). This task requires specific numerical bifurcation thresholds not recoverable from logical rules alone. Two models (LLaMA 3.3-70B and Mistral-7B) exhibited negative MCC on this task, indicating systematic anti-correlation.
- FOL Inference: Models perform significantly better (MCC = 0.52) when premises are explicitly provided.
- Indicator Diagnostic: Qwen 2.5-32B dominates this category (MCC = 0.91), outperforming all proprietary models.
Proprietary vs. Open-Source: The advantage of proprietary models concentrates on cross-indicator reasoning (+0.40 advantage) and consistency tasks. Open-source models show near-parity on formal deduction and perturbation robustness.
Prediction Bias: Models like LLaMA 3.1-8B exhibit prior collapse, predicting TRUE only 21.6% of the time (vs. 49.5% ground truth) to achieve high True Negative Rates, inflating accuracy while yielding low MCC.
MaxSAT Repair: Applying a MaxSAT post-processor to fix axiom inconsistencies improved MCC for weak models on atomic questions (e.g., +0.11 for LLaMA 3.1-8B) but degraded performance on compositional families (e.g., multi-hop), suggesting that compositional errors stem from reasoning gaps rather than simple axiom violations.
Chain-of-Thought (CoT): Preliminary CoT testing on hard families with LLaMA 3.1-8B resulted in a high invalid rate (unparseable outputs) and did not improve reasoning on regime transition questions.

Significance and Claims

The paper claims to reveal three specific pathologies in current LLM reasoning:

Prior Collapse: Models achieve moderate accuracy by defaulting to class priors (e.g., always predicting FALSE) rather than reasoning.
Surface-Form Fragility: Models fail consistency checks when facts are rephrased, indicating fragile retrieval mechanisms.
Inability to Reason on Parameter-Dependent Dynamics: Models cannot supply the quantitative premises (bifurcation thresholds) required for regime transition reasoning, even when they can perform logical deduction on given premises.

The authors identify a knowledge-type boundary: LLMs can learn logical rule-following from text (FOL inference), but they struggle with parameter-dependent dynamics that require numerical grounding. This gap does not close with model scale (e.g., scaling Qwen from 7B to 32B improved multi-hop reasoning but left regime transition near-random). The work suggests that hybrid approaches coupling LLMs with numerical solvers may be necessary to bridge the compositional gap, while noting that current CoT prompting may introduce format-compliance issues without solving underlying reasoning deficits.