A Dual-Task Paradigm to Investigate Sentence Comprehension Strategies in Language Models

1. Problem Statement

While Large Language Models (LMs) have demonstrated human-like behaviors when their cognitive resources are restricted (e.g., predicting reading times), it remains unclear whether these constraints alter their sentence comprehension strategies. Specifically, do LMs, like humans, shift toward rational inference (prioritizing semantic plausibility over strict grammatical structure) when under high cognitive load?

Existing methods for constraining LMs (e.g., truncating input length or reducing model parameters) suffer from methodological limitations:

• They often fail to replicate the dynamic balance between memory storage and processing that characterizes human working memory.
• They typically compare different model configurations rather than observing how a single model adapts its strategy under varying conditions.

2. Methodology

2.1 The Dual-Task Paradigm

The authors propose a novel experimental paradigm inspired by human psycholinguistic Operation Span tasks. In this setup, LMs must simultaneously perform two tasks:

1. Sentence Comprehension: Answering Yes/No questions about a sentence.
2. Arithmetic Computation: Solving arithmetic problems interleaved within the sentence.

The input format interleaves words and math problems (e.g., "The 2 cocktail + blended 3 = ..."). The models are tested under three conditions:

• Single Task: Read a sentence and answer a question (No math).
• Noisy Single Task: Read a sentence with embedded math but ignore the math and answer the question.
• Dual Task: Read the sentence, solve the math problems, and then answer the comprehension question.

2.2 Dataset and Stimuli

• Source: A subset of the GELP dataset (Asami and Sugawara, 2024).
• Variables:
  • Plausibility: Sentences are either Plausible (e.g., "The bartender blended the cocktail") or Implausible (e.g., "The cocktail blended the bartender").
  • Construction: Six syntactic structures were tested (Transitive, Passive, Dative, Experiencer Subject, Experiencer Object, Benefactive).
  • Arithmetic: Ten types of addition problems varying in digit length (1, 3, 5, 10, 30 digits) and number of addends (2 or 3).
• Total Data: 2,560 sentence-question pairs.

2.3 Models and Human Participants

• LMs Evaluated: GPT-4o, o3-mini, o4-mini, GPT-4.1, DeepSeek-V3, Llama-3.3, and Gemma-3.
  • Selection Criteria: Models were filtered to ensure >70% accuracy on Single Task implausible sentences and >80% accuracy on 1-digit arithmetic in the Dual Task.
• Human Experiment: 33 native English speakers recruited via Prolific. They performed the same tasks (limited to 1-digit arithmetic due to difficulty), providing a baseline for human cognitive load effects.

2.4 Evaluation Metrics

The study uses a difference-in-differences approach to measure the "plausibility effect" (the accuracy gap between plausible and implausible sentences).

• Hypothesis: If a model adopts rational inference under load, the accuracy gap between plausible and implausible sentences should widen in the Dual Task compared to the Single/Noisy tasks. This is because the model will fail more often on implausible sentences (misinterpreting them as plausible) while maintaining high accuracy on plausible ones.

3. Key Results

3.1 Shift Toward Rational Inference

• GPT-4o, o3-mini, and o4-mini exhibited a significant shift toward rational inference under the Dual Task condition.
  • These models showed a larger accuracy gap between plausible and implausible sentences in the Dual Task compared to Single/Noisy tasks.
  • Specifically, their accuracy on implausible sentences dropped significantly in the Dual Task, while accuracy on plausible sentences remained high. This indicates they began interpreting implausible sentences based on world knowledge (e.g., treating "The cocktail blended the bartender" as "The bartender blended the cocktail") rather than syntactic structure.
• Other Models (GPT-4.1, DeepSeek-V3, Llama-3.3, Gemma-3): Showed reduced accuracy in both Noisy and Dual tasks but did not show a distinct increase in the plausibility gap, suggesting they relied on different failure modes or did not shift strategies as strongly.

3.2 Specific Error Patterns

• Syntactic Vulnerability: The models most prone to rational inference errors (GPT-4o, o3-mini, o4-mini) struggled most with Passive, Dative, and Benefactive constructions.
  • Reasoning: When function words (e.g., "by," "to," "for") are ignored due to cognitive load, the remaining content words often form a semantically plausible sequence (e.g., "bartender blend cocktail"). The models rely on superficial word order and world knowledge rather than function words.
• Acquiescence Bias: Errors were more frequent when the correct answer was "No." Models tended to answer "Yes" (acquiescence), mirroring human "yea-saying" bias under cognitive load.
• Cognitive Load Scaling: Error rates increased with the complexity of the arithmetic (e.g., 30-digit problems), confirming that the effect is driven by resource depletion.

3.3 Human Comparison

Human participants showed the same pattern: lower overall accuracy and a significantly larger plausibility gap in the Dual Task (0.064 difference) compared to Single (0.025) and Noisy (0.035) tasks. This validates that the LMs' behavior mirrors human cognitive limitations.

4. Key Contributions

1. Novel Paradigm: Introduced a Dual-Task Paradigm that effectively constrains the balance between memory storage and processing in LMs, mimicking human working memory more accurately than previous length-based or parameter-based constraints.
2. Evidence of Rational Inference: Demonstrated that specific advanced LMs (GPT-4o, o3-mini, o4-mini) shift toward rational inference (prioritizing plausibility over syntax) when cognitive resources are taxed, a behavior previously observed primarily in humans.
3. Mechanism Identification: Identified that under load, models rely on superficial word sequences and world knowledge while neglecting function words, particularly in complex syntactic structures like passives and datives.
4. Theoretical Extension: Extended the understanding of human-like language comprehension from "reading costs" (processing speed) to "reading strategies" (interpretation logic), supporting the view that resource allocation is fundamental to human-like comprehension.

5. Significance and Implications

• Cognitive Modeling: The study provides strong evidence that LMs are not just statistical pattern matchers but utilize mechanisms analogous to human working memory. When these resources are strained, they exhibit the same "good-enough" processing strategies as humans.
• Safety and Reliability: The findings suggest that under high cognitive load (e.g., complex multi-step reasoning), LMs may degrade their syntactic processing capabilities, leading to hallucinations where they force semantically plausible but factually incorrect interpretations.
• Future Research: The paradigm offers a robust tool for investigating the internal mechanisms of LMs (e.g., attention mechanisms as working memory) and for conducting more rigorous human-LM comparisons.

Conclusion

The paper concludes that task-induced cognitive load, rather than just input length, is a critical factor in constraining LMs. By forcing models to balance memory storage (solving math) with processing (comprehending sentences), the study reveals that advanced LMs adopt human-like rational inference strategies, prioritizing semantic plausibility when their cognitive resources are insufficient to maintain strict syntactic analysis.