Prompt Architecture Determines Reasoning Quality: A Variable Isolation Study on the Car Wash Problem

1. Problem Statement

The study addresses a specific reasoning failure in Large Language Models (LLMs) known as the "Car Wash Problem."

• The Prompt: "I want to wash my car. The car wash is [X] meters away. Should I walk or drive?"
• The Failure: Major models (Claude, GPT-4, Gemini) consistently recommend walking.
• The Correct Answer: Drive. The implicit physical constraint is that the car must be present at the car wash to be washed; therefore, the car must be driven there.
• The Core Issue: LLMs fail to infer unstated prerequisites (the "Frame Problem") and default to distance-heuristics (100m is a short walk) rather than analyzing the goal state.

The specific research question was not why models fail (a topic covered in prior discourse), but which specific layer of a prompt architecture resolves this failure within a single model. This was motivated by the authors' observation that their "InterviewMate" system (using a complex prompt stack) solved the problem, while standalone models failed, but the specific contributing layer was unknown.

2. Methodology

The study employed a variable isolation experimental design using Claude Sonnet 4.5 (version claude-sonnet-4-5-20250929).

• Experimental Setup:

  • Total Trials: 120 API calls (20 trials per condition).
  • Hyperparameters: Temperature 0.7 (to introduce variance and measure probabilistic pass rates), max_tokens 512.
  • Prompt Modification: The distance was set to 100 meters (per the ryan-allen/car-wash-evals benchmark) to increase ambiguity compared to the original 50m.
  • Scoring: Intent-based pattern matching (regex) to distinguish between "walk" and "drive" recommendations, rather than simple word matching.
  • Recovery Test: A follow-up "challenge prompt" ("How will I get my car washed if I am walking?") was used to measure self-correction capabilities.

• Six Experimental Conditions:

  1. Condition A (Bare): No system prompt.
  2. Condition B (Role Only): Expert advisor persona injected.
  3. Condition C (Role + STAR): Added a STAR framework (Situation, Task, Action, Result), forcing the model to articulate the goal before reasoning.
  4. Condition D (Role + Profile): Injected physical user context (name, car model, location, parking status) without STAR.
  5. Condition E (Full Stack): Role + STAR + Profile + RAG (simulated context retrieval).
  6. Condition F (Role + STAR + Profile): Added post-hoc to isolate the marginal contribution of Profile vs. RAG.

3. Key Results

Primary Pass Rates (First Response)

Key Findings

1. Structured Reasoning > Context Injection:
   • STAR (Condition C) achieved 85% accuracy.
   • Profile Injection (Condition D) achieved only 30% accuracy.
   • Statistical Significance: The difference is highly significant (Fisher's exact test, $p = 0.001$, odds ratio = 13.22). Structured reasoning outperformed direct context injection by a factor of 2.83×.
2. Marginal Contributions:
   • STAR alone: +85 percentage points (pp) over baseline.
   • Profile on top of STAR: +10 pp (85% $\to$ 95%).
   • RAG on top of STAR + Profile: +5 pp (95% $\to$ 100%).
   • Conclusion: Profile data is twice as effective as RAG context in the final stretch, but both are negligible without the STAR framework.
3. Role Definition is Ineffective: Adding a persona (Condition B) without reasoning structure resulted in 0% pass rate, identical to the bare baseline.
4. The Recovery Paradox:
   • Models that failed with unstructured prompts (A, B, D) had high recovery rates (95–100%) when challenged.
   • Models that failed with structured prompts (C, F) had low recovery rates (67% and 0%, respectively).
   • Mechanism: Once a model generates a coherent, structured argument for the wrong answer, the autoregressive nature of the model makes it difficult to contradict its own prior "Task" formulation.

4. Mechanism of Action: Why STAR Works

The paper identifies the "Task" step in the STAR framework as the critical mechanism.

• Without STAR: The model jumps from "100 meters" $\to$ "Distance Heuristic" $\to$ "Walk." The goal (washing the car) is ignored.
• With STAR: The model is forced to explicitly define the Task before reasoning.
  • Correct Path: "Task: Get the car to the car wash." (The car becomes the subject, forcing the "Drive" conclusion).
  • Incorrect Path: "Task: Get yourself to the car wash." (The person becomes the subject, allowing "Walk").
• Insight: STAR does not provide new information; it forces the model to externalize implicit constraints into the context window before generating the conclusion. This changes the probability distribution of subsequent tokens.

5. Significance and Implications

• Prompt Engineering Priority: The study challenges the common industry practice of solving reasoning failures by simply adding more context (RAG, user profiles, more facts). The authors argue that how the model processes information (structure) is more critical than how much information it receives.
• Goal Articulation: The ability to explicitly state the goal ("Task") acts as a gatekeeper for implicit physical constraints.
• Latency Trade-off: Structured reasoning (STAR) increases median latency by ~69% (from ~4.6s to ~7.8s), but this overhead yields a massive accuracy gain.
• Future Work: The study highlights the need for mechanistic interpretability research to understand which attention heads activate during the "Task" formulation step and whether these findings generalize across different model architectures (e.g., GPT-4, Gemini).

Conclusion

The paper concludes that Prompt Architecture Determines Reasoning Quality. The "Car Wash Problem" is solved not by feeding the model more data about the car or the user, but by forcing the model to articulate the Task explicitly. The STAR framework accounts for the vast majority of the performance lift (+85pp), proving that structural constraints on the reasoning process are the primary driver of success in implicit constraint reasoning tasks.