CAKE: Cloud Architecture Knowledge Evaluation of Large Language Models

This paper introduces CAKE, a novel benchmark comprising 188 expert-validated questions across four cognitive levels and five cloud-native topics, to evaluate and reveal how different response formats and model configurations influence the assessment of large language models' understanding of cloud architecture.

The Gist

Imagine you are hiring a new architect to design a massive, futuristic city made entirely of digital clouds. You have two types of candidates: tiny, eager apprentices (small AI models) and seasoned, master builders (large AI models).

The paper you shared, CAKE, is essentially a rigorous job interview designed to test how well these AI "architects" actually understand the complex rules of building cloud cities.

Here is the story of their findings, broken down into simple concepts:

1. The Problem: The "Fake Expert" Trap

Until now, we've tested AI on coding tasks (like asking them to write a specific line of code) or general trivia. But asking an AI to design a cloud system is different. It's like asking someone to write a sentence versus asking them to design a bridge.

The researchers realized there was no test to see if an AI truly understood the concepts behind cloud architecture (like how to make a system resilient or how to break a big app into smaller pieces). They needed a way to see if the AI was a genuine expert or just a "parrot" repeating facts it memorized.

2. The Solution: The CAKE Exam

They created CAKE (Cloud Architecture Knowledge Evaluation). Think of this as a multi-level driving test for AI architects.

• The Questions: They built 188 questions, vetted by human experts.
• The Levels (Bloom's Taxonomy): The test isn't just one thing. It has four levels of difficulty, like climbing a ladder:
  1. Recall: "What is a container?" (Simple memory).
  2. Analyze: "Why did this system crash?" (Understanding relationships).
  3. Design: "Draw me a system that won't crash." (Creating new solutions).
  4. Implement: "Write the actual plan to build it." (Doing the hard work).
• The Topics: They covered five key areas: Patterns, Quality, Breaking things down, Cloud deployment, and "Technical Debt" (the mess you leave behind for later).

3. The Test Subjects

They didn't just test one AI. They tested 22 different versions of AI models, ranging from tiny ones (0.5 billion "brain cells" or parameters) to massive ones (70 billion). They also tested them in three "modes":

• Base Mode: Just answer the question.
• Think Mode (+think): "Take a deep breath and think step-by-step before answering."
• Tool Mode (+tool): "Go search the internet and use tools to help you."

4. The Big Surprises (The Results)

🍰 The "Multiple Choice" Ceiling

When the AI took the Multiple Choice (MCQ) part of the test, the results were surprisingly boring for the big models.

• The Analogy: Imagine a trivia game. Once a student has studied enough (about 3 billion parameters), they get almost 100% of the answers right.
• The Finding: Whether the AI was a 3-billion-parameter model or a 70-billion-parameter giant, they both scored near-perfectly on multiple-choice questions. The test couldn't tell the difference between a good student and a genius because the questions were too easy for the big ones.

📝 The "Free Response" Reality Check

Then, they asked the AI to write out their own answers (Free Response).

• The Analogy: This is like asking the student to explain their reasoning or draw the blueprint, rather than just circling "A" or "B."
• The Finding: Here, the differences exploded. The tiny models struggled to explain why they chose a design, while the big models wrote clear, logical plans.
• Key Insight: Multiple-choice questions lie. They make small models look smarter than they are. Only free-response questions reveal who can actually do the work.

🧠 The "Thinking" and "Tools" Twist

• Thinking (+think): Telling the AI to "think step-by-step" helped the small models write better essays (free response), but sometimes confused them on multiple-choice questions. It's like a student over-analyzing a simple math problem and getting it wrong.
• Tools (+tool): Giving the AI internet access was a disaster for the tiny models. They got lost in the search results and gave worse answers. It's like giving a compass to a toddler; they just spin in circles. The AI needed to be at least a certain size (around 8 billion parameters) before it could use tools effectively.

5. The "Conviction" Meter

One clever trick the researchers used was asking the AI the same question three times.

• If the AI gave the same answer all three times, they called it "High Conviction." These answers were right 90% of the time.
• If the AI changed its mind between runs, it was "Low Conviction." These answers were only right 55% of the time.
• Takeaway: If an AI seems unsure (changes its answer), humans should double-check its work.

6. The Final Verdict

The paper concludes with a warning for anyone using AI in software architecture:

Don't trust the multiple-choice scores.

If you ask an AI to pick the right answer from a list, even a small, cheap AI will look like a genius. But if you ask it to design a system or explain a complex decision, you need a much larger, smarter model.

In short:

• Small Models: Good at memorizing facts and picking the right letter on a test.
• Big Models: Good at actually designing and explaining complex systems.
• The Test: You must ask them to write, not just pick, to know what they can really do.

The researchers made their "exam questions" (the CAKE dataset) public so everyone can keep testing and improving these digital architects.

Technical Summary

1. Problem Statement

While Large Language Models (LLMs) are increasingly used as co-pilots in software engineering, there is a critical gap in evaluating their understanding of cloud-native software architecture. Existing benchmarks (e.g., SWE-bench, HumanEval) focus on code generation or reasoning, while broad knowledge benchmarks (e.g., MMLU) neglect the specific domain of software architecture.

Current evaluations fail to assess whether LLMs possess the conceptual and procedural knowledge required to make architectural decisions regarding microservices, containerization, orchestration, and cloud deployment patterns. Practitioners lack a reliable metric to determine where LLM assistance is trustworthy versus where human judgment remains essential.

2. Methodology: The CAKE Benchmark

The authors introduce CAKE, a specialized benchmark designed to evaluate LLMs across four cognitive levels of Bloom's revised taxonomy and five cloud-native topics.

• Dataset Composition:

  • Total Questions: 188 expert-validated questions (130 Multiple Choice Questions [MCQ] and 58 Free-Response [FR]).
  • Cognitive Levels: Recall, Analyze, Design, and Implement.
  • Topics: Architectural Patterns, Quality Attributes, Decomposition Strategies, Cloud Deployment, and Technical Debt.
  • Validation: Questions were generated by Claude Opus 4.5 and rated by four domain experts on clarity, correctness, and difficulty. The final set achieved high mean scores (Clarity: 4.72/5; Correctness: 4.63/5).

• Models Evaluated:

  • Scope: 22 model configurations across four families (Qwen, Llama, Mistral, GPT).
  • Scale: Ranging from 0.5B to 70B parameters.
  • Configurations: Evaluated in three modes:
    1. Base: Direct question answering.
    2. +think: Structured reasoning (Chain-of-Thought).
    3. +tool: Agentic tool use (web search/browsing).

• Evaluation Pipeline:

  • MCQs: Evaluated using three-run majority voting to reduce positional bias and measure "conviction" (consistency).
  • Free-Response: Evaluated using LLM-as-a-Judge (DeepSeek-R1:32B) with a deterministic 0–5 scoring rubric.

3. Key Contributions

1. CAKE Benchmark: The first benchmark specifically targeting cloud-native architectural knowledge across multiple cognitive levels, including a unique focus on "Implementation" via free-response.
2. Empirical Analysis: A comprehensive evaluation of 22 model configurations, providing data on how parameter size, reasoning augmentation, and tool use affect architectural competence.
3. Public Artifacts: The release of the benchmark dataset and evaluation pipeline to the community.

4. Key Results & Findings

A. MCQ Performance (Knowledge Recall)

• Performance Ceiling: MCQ accuracy plateaus rapidly. Models with >3B parameters consistently achieve >90% accuracy, with top models (e.g., GPT-5-Mini, Llama 70B) reaching 99.2%.
• Diminishing Returns: Increasing parameters beyond 3B yields negligible gains in MCQ accuracy (1–3 percentage points).
• Confidence Signal: "Conviction" (unanimous agreement across three runs) is a strong predictor of accuracy. Unanimous answers achieved 89.5% accuracy, whereas split-majority answers dropped to 55.0%.

B. Free-Response Performance (Generative Competence)

• Differentiation: Unlike MCQs, free-response scores scale steadily across the entire parameter range (0.5B to 70B), effectively distinguishing model capabilities.
• Implementation Gap: The "Implement" level (procedural knowledge) showed the widest performance gap (1.36 to 4.54 out of 5), highlighting that generating architectural solutions is significantly harder than selecting them.
• Family Variance: Training data composition matters. For instance, Mistral 3B outperformed larger models (Qwen 7B, Llama 8B) in free-response tasks, suggesting architecture knowledge depends on data quality, not just parameter count.

C. Augmentation Effects

• Reasoning (+think):
  • Small Models: Significantly improves performance (e.g., Llama 1B MCQ accuracy nearly doubled from 37.7% to 76.2%).
  • Larger Models: Can degrade MCQ performance due to "over-thinking" (e.g., Mistral 3B dropped 13.1 pp in MCQ accuracy) while slightly improving free-response quality.
• Tool Use (+tool):
  • Negative Impact on Small Models: Caused severe performance degradation for models <8B parameters (e.g., Llama 3B dropped 53 pp in MCQ).
  • Threshold: Effective tool use requires a minimum capacity of ~8B parameters. Below this, models fail to invoke tools correctly or get stuck in loops.

5. Significance and Implications

• Evaluation Format Matters: Relying solely on MCQs overestimates LLM capabilities in software architecture. MCQs saturate quickly, while free-response reveals the true depth of architectural reasoning and implementation skills.
• Practitioner Guidance:
  • Model Selection: For architectural design and implementation, practitioners should prioritize free-response evaluation over MCQ accuracy.
  • Confidence Filtering: The "conviction metric" (unanimous voting) can be used in production to flag low-confidence architectural suggestions for human review.
  • Augmentation Strategy: Reasoning augmentation is beneficial for small models but risky for larger ones in MCQ contexts. Tool augmentation should be restricted to models with >8B parameters.
• Educational Impact: The benchmark maps directly to Bloom's taxonomy, helping educators and engineers understand which cognitive tasks (Recall/Analyze vs. Design/Implement) can be delegated to AI.

Conclusion

CAKE demonstrates that while LLMs have mastered the recall of cloud-native concepts (MCQs), their ability to design and implement architectures (Free-Response) varies significantly by model size and training data. The study advocates for a shift in benchmarking from multiple-choice selection to generative evaluation to accurately measure architectural competence.