Beyond Models: Reflections on Engineering AI-enabled Systems in a Project-Based Course

Technical Summary: Beyond Models: Reflections on Engineering AI-enabled Systems in a Project-Based Course

Problem Statement
The integration of Artificial Intelligence (AI) into software systems has shifted the engineering challenge from isolated model development to the construction of complex, data-driven systems. As noted by Sculley et al. [1], real-world AI systems are entangled with code, data, configuration, and infrastructure, creating "hidden technical debt" that extends beyond model accuracy. While machine learning education often emphasizes algorithmic design, students frequently lack experience in the architectural design, deployment, monitoring, and lifecycle management of AI-enabled systems. Existing educational approaches often fail to bridge the gap between theoretical ML concepts and the system-level software engineering (SE) practices required for robust, scalable, and maintainable AI applications. This paper addresses the need for educational frameworks that expose graduate students to these systemic challenges through project-based learning.

Methodology
The authors conducted a mixed-methods study based on the design and implementation of a master's-level course titled AI Algorithms: Theory and Engineering at the University of Bremen. The course utilized a project-based learning model where 26 students, organized into seven teams, developed an end-to-end movie recommendation system.

The study's data collection and analysis involved:

1. Course Structure: The curriculum featured five scaffolded assignments leading to a final project. These assignments progressively covered initial prototyping, requirements engineering, architectural design (including monolithic vs. microservices), trade-off analysis using the Architecture Trade-off Analysis Method (ATAM), and adaptation to evolving requirements.
2. Data Sources:
   • Student Artifacts: Analysis of team-based submissions, including analytical reports, architectural designs, technical documentation, and implemented code.
   • Questionnaire: A survey administered to 14 participating students (out of 26 registered) covering their background, reflections on learning outcomes, and course evaluations.
3. Analysis Approach: The researchers employed thematic analysis [14] to identify patterns in qualitative data from both the questionnaire and the feedback provided on student artifacts. This approach was used to address four specific Research Questions (RQs) regarding student challenges, the effectiveness of bridging theory and practice, student takeaways, and areas for course improvement.

Key Contributions
The paper makes three primary contributions to the field of Software Engineering for AI (SE4AI) education:

1. Course Design: The operationalization of a project-based master's course that explicitly addresses the gap between ML theory and system-level SE practice. The course structure moves students from isolated model development to building scalable, fault-tolerant AI-enabled systems using contemporary tools (e.g., Docker, Kafka, Prometheus, Flask).
2. Empirical Analysis: A detailed analysis of student assignments, final projects, and survey responses that identifies common challenges, recurring mistakes, and the specific learning outcomes achieved. This includes an examination of how students handle heterogeneous ML integration, data management, and evolving requirements.
3. Educational Recommendations: A set of derived recommendations for educators to inform the design and continuous improvement of similar courses, focusing on feedback mechanisms, assignment clarity, and the integration of practical workshops.

Results
The study identified several critical findings regarding the challenges and learning outcomes of engineering AI-enabled systems:

• Common Challenges (RQ1 & RQ1.1):

  • Knowledge Gaps: Students faced significant difficulties due to uneven expertise in ML and SE. Beginners struggled with the steep learning curve of ML concepts and DevOps tools (e.g., Kafka, CI/CD), while communication breakdowns occurred within teams with varying skill levels.
  • Architectural Decisions: Students struggled with early architectural decisions, particularly regarding model switching mechanisms, fallback strategies, and managing dependencies. Many designs lacked the modularity required to handle evolving requirements.
  • Data and Integration: Practical challenges centered on data quality (missing values, noise), model robustness, and the complexity of integrating heterogeneous modules. Changes in requirements often necessitated major system overhauls, including database re-engineering and model retraining.
  • Team Dynamics: Issues such as disengaged team members and biased decision-making impacted project success and morale.

• Bridging Theory and Practice (RQ2):

  • The course successfully facilitated a transition from theoretical understanding to practical implementation. Students reported that the project exposed them to the reality that "machine learning is not about constructing models; it's about data and preprocessing."
  • Exposure to evolving requirements (via Assignment 5) forced students to engage in architectural revision and trade-off evaluation, simulating real-world uncertainty.
  • Integration complexity served as a meaningful learning opportunity, revealing that building functional AI systems involves managing deployment pipelines, monitoring tools, and system-level performance, not just model training.

• Student Takeaways (RQ3):

  • Data-Centricity: Students shifted their perspective from model-centric to data-centric, recognizing that data quality and evaluation strategies often outweigh algorithmic choice.
  • System-Level Thinking: Students developed an appreciation for scalable, modular architectures and the necessity of structured development processes (e.g., ATAM, requirements engineering) to manage system evolution.
  • Tool Proficiency: The course enhanced technical literacy in diverse toolchains (Docker, Kubernetes, Flask, etc.), preparing students for professional system development.

• Areas for Improvement (RQ4):

  • Students advocated for retaining the project-based structure but requested more timely and constructive feedback, particularly during the design phase.
  • There was a call for clearer assignment scopes and more targeted technical workshops (e.g., on Docker and Flask integration) to support implementation.
  • Some students noted a disconnect between theoretical assignments and incremental coding, suggesting a need to couple documentation activities more closely with implementation to distribute workload evenly.

Significance and Claims
The paper modestly claims that the course design effectively fostered system-level reasoning and strengthened student awareness of data-centric ML practices. The authors assert that the study provides valuable insights for educators aiming to integrate AI and software engineering education, highlighting the necessity of moving beyond isolated model development to holistic system engineering.

The authors acknowledge limitations in external validity due to the study being conducted in a single course with a small cohort at one university. Consequently, they caution against direct generalization of the findings but suggest that the identified patterns and challenges can inform the design of similar courses. The study emphasizes that while the course met its learning goals, continuous refinement based on student feedback and artifact analysis is essential for improving the educational experience in SE4AI.