Technical Summary: Less Back-and-Forth: A Comparative Study of Structured Prompting

Problem Statement

Large Language Models (LLMs) are increasingly deployed for open-ended tasks such as summarization, planning, explanation, and coding. However, underspecified or vague prompts often lead to low-quality responses, necessitating additional user interaction to correct errors or refine outputs. This "back-and-forth" increases user effort and interaction cost. While it is informally understood that better prompts yield better results, there is a practical gap in evidence regarding the comparative efficacy of different prompting strategies. Specifically, it remains unclear whether the benefits of simply rewriting a prompt (structured prompting) outweigh or differ from interactive clarification approaches (where the model asks questions first), and whether these effects are consistent across different task types and LLM architectures.

Methodology

The study employs a small-scale comparative design to evaluate three distinct prompt conditions across four task categories (summarization, planning, explanation, and coding) and three LLM systems (ChatGPT, Claude, and Grok).

Experimental Conditions

1. Raw Prompt: A baseline condition where a human writes a short, direct prompt as a typical user might, intentionally omitting specific details, roles, or formatting constraints.
2. Checklist-Improved Prompt: The raw prompt is rewritten using a structured checklist comprising three elements:
   • Roles/Rules: Defining the model's persona or constraints.
   • Context: Explaining the audience and purpose of the task.
   • Answer Format: Specifying the desired output structure.
3. Clarifying-Question Prompt: The model is instructed to ask 1–3 clarifying questions before generating a final answer. The interaction continues until the user responds, after which the model provides the final output.

Evaluation Metrics

• Output Quality: Assessed using a unified rubric scoring four dimensions (Task Completion, Correctness, Compliance, Clarity) on a scale of 0–2, yielding a total score of 0–8. Scores are mapped to five interpretive levels (Complete Failure to Flawless).
• Interaction Effort: Measured by "turns-to-acceptance" (where one turn equals one user message and one model reply) and total token usage (input + output).
• Procedure: Each condition was executed in a new session to prevent context carryover. Three human operators evaluated the models (one per model) using the same rubric. Token counts were recorded using model-specific tokenizers.

Key Contributions

The paper makes three primary contributions:

1. Comparative Framework: It presents a controlled framework for comparing three common prompting strategies (raw, checklist, and interactive clarification) across multiple models and tasks.
2. Cross-Model Stability Analysis: It tests whether the efficacy of structured prompting remains stable across different LLM systems (ChatGPT, Claude, Grok) and diverse task types.
3. Dual-Metric Evaluation: It evaluates both final answer quality and interaction cost simultaneously, rather than treating them as separate concerns, providing a holistic view of the quality-effort tradeoff.

Results

The study yielded the following findings:

• Quality Performance: Checklist-improved prompts achieved the highest mean rubric score (7.50/8), significantly outperforming raw prompts (5.67/8) and clarifying-question prompts (6.67/8).
• Interaction Efficiency: Checklist prompts required an average of 1.00 turn to acceptance, matching the raw prompt baseline but with vastly superior quality. In contrast, clarifying-question prompts required an average of 1.96 turns due to the initial question-answer exchange.
• Token Usage: Checklist prompts demonstrated the best quality-effort tradeoff regarding token usage, averaging 683.42 tokens, compared to 962.25 for raw prompts and 936.50 for clarifying prompts.
• Task and Model Variability:
  • Task Types: The checklist condition showed the strongest gains in coding (improving from 4.67 to 7.83) and planning. For summarization, checklist and clarifying prompts performed equally well.
  • Model Differences: ChatGPT and Grok achieved their highest scores with checklist prompts. Claude performed slightly better with clarifying prompts than with checklists, though checklists still required fewer turns overall.
• Hypothesis Validation:
  • H1 (Checklist > Raw): Supported. Checklists produced higher-quality outputs.
  • H2 (Clarifying > Raw & Checklist): Partially supported. Clarifying improved over raw prompts but did not outperform checklist prompts overall.
  • H3 (Reduced Effort): Supported for checklists (high quality in one turn) but not for clarifying prompts (which increased interaction turns).

Significance and Claims

The paper claims that a simple prompt checklist—specifying roles, context, and answer format—can significantly improve LLM response quality while reducing unnecessary interaction. The findings suggest that structured prompting offers a superior balance between output quality and interaction effort compared to both unstructured raw prompts and interactive clarification strategies.

The authors emphasize that while "better prompts" are often assumed to be beneficial, this study provides empirical evidence that checklist-improved prompts are a more efficient strategy than relying on the model to ask clarifying questions. The study concludes that even basic structural guidance can make model responses more usable and acceptable in a single turn, thereby reducing the "back-and-forth" typical of human-AI interaction.

Limitations

The authors acknowledge several constraints:

• The study is limited in scale (four tasks, three models) and should not be generalized to all prompting scenarios.
• Quality scores rely on author evaluation rather than independent blind raters or external user studies, introducing potential subjectivity.
• The benchmark tasks may not fully represent complex enterprise or real-world settings.
• Operator-specific variation may exist since each model was handled by a single human evaluator.

Future work is proposed to include external human participants, blind pairwise ratings, and ablation studies to determine which specific checklist components (role, context, or format) contribute most to the observed improvements.