🤖 AI

Five Fatal Assumptions: Why T-Shirt Sizing Systematically Fails for AI Projects

This paper argues that traditional T-shirt sizing estimation fails for AI projects due to five flawed assumptions about linearity and predictability, proposing instead a "Checkpoint Sizing" framework that uses iterative decision gates to adapt scope based on ongoing learning.

Original authors: Raja Soundaramourty, Ozkan Kilic, Ramu Chenchaiah

Published 2026-02-23

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Raja Soundaramourty, Ozkan Kilic, Ramu Chenchaiah

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a contractor hired to build a house. You've built hundreds of houses before. You know that a small cottage takes two weeks, a medium house takes four, and a mansion takes eight. You use a simple "T-shirt sizing" method: you look at the blueprints, say "That's a Medium," and give your client a price and a date. This works great for houses because houses are predictable. If you double the number of rooms, you roughly double the work.

Now, imagine your client asks you to build a living, breathing house that learns, talks back, and changes its own mind every time someone walks through the door. This is an AI project.

The authors of this paper, engineers from Cisco, are saying: "Stop trying to estimate AI projects like you're building a house. The rules are completely different, and your old T-shirt sizing method is going to get you fired."

Here is the breakdown of the Five Fatal Assumptions that make T-shirt sizing fail for AI, explained with simple analogies.

1. The "Linear Ladder" Fallacy

The Old Way: In normal software, if you want to make a feature twice as big, it takes twice as much work. It's like climbing a ladder: step 1, step 2, step 3.
The AI Reality: AI is more like climbing a mountain where the air gets thinner the higher you go.

The Analogy: Getting a model 85% accurate is easy. But getting it from 85% to 95%? That might require 10 times more data, 10 times more computer power, and 10 times more trial and error.
The Trap: You think adding one more "smart feature" is a small step. In AI, that one step might require a massive, exponential jump in resources. The "last mile" of AI isn't a mile; it's a marathon.

2. The "Copy-Paste" Illusion

The Old Way: If you built a chatbot for a bank last year, you think you know exactly how long it will take to build one for a hospital. You rely on past experience.
The AI Reality: Every dataset is a new, uncharted island.

The Analogy: Building a chatbot for a bank is like navigating a calm river. Building one for a hospital is like navigating a stormy ocean with different currents, different monsters, and different weather.
The Trap: You think you can reuse your "map" from the bank project. But in AI, the data is different, the slang is different, and the edge cases are different. You will run into "unknown unknowns" (problems you didn't even know existed) that only show up when you start training the model.

3. The "More Hands" Myth

The Old Way: If a task takes 8 weeks with one person, you think it will take 4 weeks with two people. You can just add more workers to speed things up.
The AI Reality: AI development has traffic jams that you can't fix by adding more cars.

The Analogy: Imagine a relay race where the baton must be passed in a specific order: First, you must gather the data. Then you can clean it. Then you can train the model. Then you can test it.
The Trap: You can't hire 10 people to "gather data" faster if the data doesn't exist yet. You can't hire 10 people to "train the model" faster if the computer takes 3 days to run the simulation. Some steps are strictly sequential; adding more people just creates a crowd waiting in line, not faster progress.

4. The "Lego Block" Trap

The Old Way: In normal software, you can build the kitchen, the bedroom, and the bathroom separately, then snap them together at the end.
The AI Reality: AI is a tightly woven spiderweb.

The Analogy: In AI, the data, the model, and the prompts are all glued together. If you change the data (the web's anchor), the whole model (the web) shifts. If you tweak the prompt, the whole system breaks.
The Trap: You try to size the "UI team" and the "AI team" separately. But because they are so tightly coupled, a small change in one team's work causes a massive ripple effect that breaks the other team's work. You can't estimate them in isolation.

5. The "Finish Line" Mirage

The Old Way: In software, "Done" means the code compiles and the tests pass. Once it's done, it's done.
The AI Reality: In AI, "Done" is a moving target.

The Analogy: Imagine you finish a race, but the finish line keeps moving forward every time you run.
The Trap: You might think the project is "Medium" and finished. But then, a safety check reveals the AI is being biased. Or, in a long conversation, the AI starts hallucinating (making things up). Suddenly, your "finished" project needs weeks of extra work to fix safety, ethics, and reliability. The definition of "Done" keeps changing based on how the AI behaves in the real world.

The Solution: "Checkpoint Sizing"

Since we can't predict the future with a T-shirt size, the authors propose a new method called Checkpoint Sizing.

The Analogy: Instead of guessing the whole trip at the start, think of it like hiking a mountain with a series of base camps.

Don't guess the whole summit: You don't say, "We will finish in 6 months."
Set up Base Camps: You plan small checkpoints (Gate A, Gate B, Gate C).
- Gate A: "Is our data ready?" (If no, stop and fix data).
- Gate B: "Does the model actually work?" (If no, don't proceed).
- Gate C: "Is it safe?" (If no, fix safety).
Re-evaluate at every camp: At each checkpoint, you look at what you actually learned, not what you assumed. You then decide: Do we keep going? Do we change direction? Or do we stop because it's too hard?

The Bottom Line

T-shirt sizing assumes the world is predictable and linear. AI is unpredictable and chaotic.

Old Mindset: "Let's guess the size and hope for the best."
New Mindset: "Let's take small steps, check the ground at every turn, and only commit to the next step once we know it's safe."

If you try to estimate an AI project like a standard software project, you will likely end up with a project that is months late, over budget, and full of surprises. Use the "Checkpoint" method instead to navigate the unknown.

1. Problem Statement

The paper addresses the systematic failure of traditional Agile estimation techniques, specifically T-shirt sizing (categorizing work as S, M, L, XL), when applied to Artificial Intelligence (AI) projects. While T-shirt sizing works well for deterministic software development by relying on relative complexity and historical analogies, it produces misleading estimates for AI initiatives involving Large Language Models (LLMs), multi-agent systems, and Retrieval-Augmented Generation (RAG).

The core issue is that AI development is driven by model behavior and data-dependent uncertainty rather than deterministic code. Consequently, projects often experience massive scope creep, technical "walls," and timeline overruns that cannot be explained by poor execution but are inherent properties of the AI problem space. The authors argue that the mental models used for traditional software do not fit the non-linear, probabilistic nature of AI.

2. Methodology

The authors employed a structured synthesis approach to identify and validate why traditional estimation fails:

Assumption Identification: They reviewed canonical agile estimation methods (Story Points, Planning Poker, T-shirt sizing) to extract their implicit premises.
Workflow Mismatch Analysis: They mapped these premises against established characteristics of ML/AI workflows (e.g., data dependency, experimental iteration, technical debt).
Empirical Anchoring: They grounded their analysis in recent empirical literature (including arXiv preprints) focusing on multi-agent system failures, scaling laws, and LLM behavior.
Evaluation Criteria: Each assumption was tested against three criteria:
1. Validity in Traditional Software.
2. Systematic violation in AI contexts.
3. Material impact on estimation accuracy.
Evidence Collection: They utilized quantitative metrics from cited studies, such as interaction complexity growth ( $N(N-1)$ ) and performance degradation rates in multi-turn conversations.

3. Key Contributions: The Five Fatal Assumptions

The paper identifies five foundational assumptions of T-shirt sizing that collapse in AI contexts:

Assumption 1: Linear Effort Scaling

Traditional View: Effort scales linearly with scope (e.g., double the features = double the work).
AI Reality: Effort follows an exponential curve. Improving accuracy from 85% to 95% often requires 10x the effort due to diminishing returns in data and compute.
Evidence: In multi-agent systems, complexity grows combinatorially ( $N(N-1)$ ). Adding agents multiplies interaction paths and error states rather than adding a constant increment.

Assumption 2: Repeatability from Prior Experience

Traditional View: Past experience with similar tasks (e.g., building a chatbot) provides a reliable baseline for future estimates.
AI Reality: Every dataset is "uncharted territory." Data distributions, edge cases, and annotation quality vary wildly.
Evidence: LLMs exhibit a 39% average performance degradation in multi-turn conversations compared to single-turn interactions due to context drift and state loss, making prior single-turn benchmarks unreliable.

Assumption 3: Effort-Duration Fungibility

Traditional View: Duration can be compressed by adding resources (e.g., 2 engineers finish in half the time).
AI Reality: AI pipelines contain irreducible sequential dependencies (Data Collection → Cleaning → Training → Evaluation). These phases have "duration floors" that cannot be compressed by headcount.
Evidence: Multi-agent pipelines have latency floors determined by inference calls; adding engineers does not reduce the time required for sequential model training or data processing.

Assumption 4: Task Decomposability

Traditional View: Large projects can be broken into parallel workstreams with manageable integration overhead.
AI Reality: AI systems exhibit tight coupling across the stack. Changes in data pipelines cascade into model retraining; schema changes invalidate prompt logic.
Evidence: Shared resource contention (e.g., finite context windows) means one agent's verbose output can starve others, creating integration risks invisible to component-level sizing.

Assumption 5: Deterministic Completion Criteria

Traditional View: "Done" is binary (tests pass, specs met).
AI Reality: Completion is probabilistic. A project may meet accuracy targets but fail safety, legal, or bias audits, turning a "Medium" task into an "XL" remediation effort overnight.
Evidence: Emergent failure modes like infinite correction loops (Agent A rejects Agent B, which retries, leading to a loop) and "guardrail oscillation" (tightening safety reduces utility) make the definition of "done" a moving target.

4. Proposed Solution: Checkpoint Sizing

The authors propose Checkpoint Sizing as an evidence-based alternative. Instead of a single upfront estimate, this methodology treats estimation as an iterative discovery process.

Core Mechanism: The project is divided into explicit Decision Gates (Checkpoints) where scope, timeline, and feasibility are reassessed based on empirical evidence gathered during development, not initial assumptions.
The Gates:
1. Data Readiness: Validating data quality, schema, and PII policies.
2. Evaluation Harness: Establishing baseline scores, gold sets, and automated regression tests.
3. Safety & Reliability: Validating hallucination rates, multi-turn stability, and termination controls.
4. Cost & Latency: Measuring throughput, token budgets, and p95 latency.
5. Operationalization: Defining monitoring, drift detection, and rollout plans.
Outcome: At each gate, the team decides to Proceed, Pivot, Reduce Scope, or Terminate. Estimates are updated based on concrete artifacts (e.g., "Data cleaning took 3 weeks") rather than analogies.

5. Results and Illustrative Case Study

The paper includes a synthetic case study ("Support Copilot with RAG") to demonstrate the efficacy of Checkpoint Sizing:

Initial T-Shirt Estimate: "Large" (6–8 weeks), based on analogy to a standard chatbot.
Checkpoint Evolution:
- Gate A (Data): Discovered 25% redacted fields and inconsistent KB versions $\rightarrow$ +3 weeks.
- Gate B (Eval): Baseline retrieval failed on long-tail SKUs; needed new rubrics $\rightarrow$ +2 weeks.
- Gate C (Safety): Multi-turn degradation and tool-call conflicts detected $\rightarrow$ +3–5 weeks.
- Gate D (Cost/Latency): Sequential tool calls exceeded SLOs $\rightarrow$ +2 weeks.
- Gate E (Ops): Required canary plans and drift monitoring $\rightarrow$ +1–2 weeks.
Final Result: The project was re-estimated as "Extra-Large" (12–16 weeks). The shift from a static guess to an evidence-driven timeline prevented a likely failure.

6. Significance

Paradigm Shift: The paper challenges the engineering community to abandon "one-size-fits-all" estimation for AI. It argues that adding buffers to bad estimates is insufficient; the estimation model itself must change.
Risk Mitigation: By introducing explicit decision gates, organizations can avoid "sunk cost" traps where teams continue to invest in unfeasible AI projects.
Management Tool: It provides a concrete framework (Checkpoints A–E) for Engineering Managers and Product Owners to manage the unique uncertainties of LLMs and multi-agent systems.
Future Direction: The authors call for further empirical validation, tooling integration (e.g., Jira plugins for Checkpoint Sizing), and domain-specific adaptations for high-stakes fields like healthcare and finance.

In conclusion, the paper asserts that T-shirt sizing fails because it assumes a linear, deterministic world, while AI development is non-linear and probabilistic. The proposed Checkpoint Sizing framework offers a rigorous, iterative approach to navigate this uncertainty.