Spreadsheet Modeling Experiments Using GPTs on Small Problem Statements and the Wall Task

1. Problem Statement

The paper addresses the emerging capability of Generative Pre-trained Transformers (GPTs) to automate the creation of industrial-quality analytical spreadsheet models. While GPTs are widely touted for their ability to assist with data analysis and formula generation, their reliability in producing reusable, error-free spreadsheet models remains unproven.

The authors define the goal as creating models that meet six criteria for "industrial quality": efficiency, readability, transferability, accuracy, reusability, and modifiability. To make this tractable for experimentation, the authors narrowed the focus to reusability, defined by five Essential Requirements for Reusability (ERFR):

1. Input Isolation: Each input value resides in a single cell.
2. Formula Usage: All calculations use cell formulas (no hard-coded numbers in calculation cells).
3. No Hardwiring: No hardwired numbers within formulas.
4. Labeling: Appropriate labels for inputs and formulas.
5. Accuracy: The calculations must be mathematically correct.

The core problem is determining whether current GPT tools can reliably generate spreadsheets meeting these criteria from natural language problem statements, and if not, what the barriers (reproducibility, confidence, workflow) are.

2. Methodology

The research followed a structured experimental approach involving screening, selection, and detailed testing.

A. Tool Selection (Screening Phase)

• Initial Pool: Five GPT extensions claiming spreadsheet modeling capabilities were selected based on high usage volume (>100k conversations), popularity (>1,000 reviews, rating >3.3), Code Interpreter functionality, and ability to generate XLSX files.
• Evaluation Criteria: The tools were judged on Conversation Quality, User Interaction, Error Handling, File Reliability, and Spreadsheet Design.
• Selection: Excel AI (EAI) by pulsrai.com was selected as the best performer. It consistently generated well-structured, downloadable Excel files with valid formulas, whereas others produced broken files or non-functional outputs.
• Configuration: The authors noted that EAI performs best when paired with ChatGPT-4o (rather than the recommended o4-mini), as the latter tends to generate images instead of files.

B. Prompt Engineering Strategy
The authors distinguished between the Problem Statement (the logic/what to calculate) and Instructions (the desired output format).

• They established a parsimonious set of three instructions to ensure ERFR compliance:
  1. "Create an Excel model."
  2. "Use cell formulas."
  3. "Provide a downloadable Excel file."

C. Experimental Design

• Small-Scale Experiments: Eight distinct problem statements were used to test three specific variables:
  1. Data vs. Parameters: Does the model handle specific numbers (Data) or variable names without values (Parameters) differently?
  2. Explicit vs. Implicit Time: How does the model handle "30 consecutive days" vs. "a month" (which implies 30 days in analytics but is ambiguous)?
  3. Known vs. Made-up Nouns: Does the model struggle with invented terminology (e.g., "zorp" instead of "apple")?
• Large-Scale Experiment: The "Wall Task" (a standard academic problem statement regarding wall construction costs) was used to test performance on a paragraph-length, multi-variable problem.

3. Key Contributions

• Definition of ERFR: The paper formalizes the "Essential Requirements for Reusability" as a standard metric for evaluating AI-generated spreadsheets.
• Prompt Structure Framework: It proposes a separation of "Problem Statement" and "Instructions" to optimize GPT output for spreadsheet generation.
• Identification of Critical Barriers: The authors identify two fundamental challenges hindering professional adoption:
  • The Problem of Confidence: How to verify the accuracy of an AI-generated model when test cases do not exist.
  • The Problem of Workflow: How to integrate imperfect AI drafts into a human-centric development process (e.g., is it faster to fix the AI draft or start over?).
• Empirical Evidence of Instability: The study provides concrete evidence that current GPT tools are "temperamental," producing non-reproducible results even with identical prompts.

4. Results

A. Small-Scale Experiments (8 Prompts)

• Successes: EAI performed well on 7 of 8 prompts. It successfully handled parameters (blank cells ready for input), made-up nouns, and explicit "30 days" phrasing.
• Failures:
  • Ambiguity Failure: When prompted with "for a month" (Problem Statement 6), the model generated a spreadsheet with hard-coded numbers in the calculation column rather than formulas. While the numbers were accurate, the model failed the ERFR criteria for reusability.
  • Reproducibility Failure: Running the same prompt (Problem Statement 6) multiple times yielded different spreadsheet structures, some with "garbage" formulas and others with correct logic, indicating a lack of consistency.
  • Context Sensitivity: The model sometimes generated a static table of numbers instead of a dynamic calculator, even when instructed to "use cell formulas."

B. Large-Scale Experiment (The Wall Task)

• Inconsistent Performance:
  • With standard instructions, the model initially produced a correct table of numbers but no cell formulas.
  • A follow-up prompt to "make it dynamic" successfully generated a high-quality, modular ERFR-compliant model.
  • However, running the Wall Task without specific instructions resulted in a file that corrupted upon opening, requiring Excel's "Recover" function, and contained a mix of accurate, inaccurate, and missing formulas.
• Observation: The tool's performance is highly volatile; a "perfect" model can be generated in one run, while a broken or non-reusable model is generated in another.

5. Significance and Conclusion

Current State of Technology
The authors conclude that while GPTs show promise for generating drafts that might reduce development time or lower skill barriers, they are not yet ready for professional use. The tools are currently unreliable for producing "industrial-quality" models without significant human intervention.

Implications for Users

• Skill Requirement: Users must possess sufficient spreadsheet modeling skills to verify formulas and logic. GPTs cannot currently replace novice modelers because the user must be able to detect and fix the AI's errors.
• Workflow Integration: The value of GPTs lies in the "rough draft" approach. If the time saved by generating a draft (even an imperfect one) plus the time to fix it is less than building from scratch, it adds value. However, this requires a new workflow where the human acts as an auditor and refactoring engineer.

Future Research Directions

• Prompt Engineering: Developing more robust instruction sets to handle ambiguity and ensure reproducibility.
• Reproducibility: Investigating why identical prompts yield different results (e.g., model versioning, time of day, system state).
• Scale: Testing on larger, real-world business problem statements rather than simple "starter tasks."
• Verification: Developing automated methods to verify the accuracy of AI-generated models without pre-existing test cases.

In summary, the paper serves as a reality check on the current state of AI in spreadsheet analytics, highlighting that while the technology is capable of generating structure, it lacks the consistency and reliability required for autonomous professional application.