💻 computer science

Restructure This: Using AI to Restructure Onboarding Documents to Reduce Cognitive Overload

This paper presents VisDoc, a GenAI-driven prototype that applies Cognitive Theory of Multimedia Learning to restructure fragmented open source onboarding documentation into task-based, multimodal units, demonstrating through empirical evaluation that this approach significantly reduces cognitive load and improves task success for newcomers.

Original authors: Zixuan Feng, Prashant Tandan, Igor Steinmacher, Marco Aurelio Gerosa, Anita Sarma

Published 2026-05-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zixuan Feng, Prashant Tandan, Igor Steinmacher, Marco Aurelio Gerosa, Anita Sarma

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Problem: The "Wall of Text" Trap

Imagine you want to join a massive, ancient library (Open Source Software) to help fix a book. But when you walk in, the librarian hands you a single, 500-page book written in tiny font. It has no table of contents, no chapters, and the instructions for "how to fix a book" are scattered randomly between recipes for soup and maps of the moon.

This is what new contributors face with Open Source Software (OSS) documentation. The paper calls this cognitive overload. It's like trying to drink from a firehose; your brain gets so full of trying to figure out where to start, what step comes next, and how everything connects, that you get frustrated, make mistakes, and eventually give up.

The researchers found that the problem isn't just that information is missing; it's that the information is presented poorly. It's too dense, too linear (reading from start to finish without knowing what matters), and relies only on text, forcing your brain to work overtime to build a mental map of the process.

The Solution: VisDoc (The "Interactive Map")

To fix this, the researchers built a tool called VisDoc. Think of VisDoc as turning that 500-page wall of text into an interactive, 3D treasure map.

Instead of reading a long story, VisDoc uses Artificial Intelligence (specifically Generative AI) to:

Break it down: It chops the giant document into small, bite-sized "tasks" (like "Fix a typo" or "Submit a change").
Connect the dots: It figures out the order of operations and draws a tree-like structure showing which tasks lead to others.
Clean it up: It removes the boring, repetitive, or confusing parts that don't help you right now.
Show, don't just tell: For tricky steps, it doesn't just write text; it generates short videos and audio narrations to walk you through the process.

The Human Safety Net: The researchers were careful. They know AI can sometimes "hallucinate" (make things up). So, VisDoc isn't fully automatic. It acts like a draft that a human maintainer (the librarian) can review, edit, and approve before it's shown to new users. It's a "Human-in-the-Loop" system.

The Theory: How Our Brains Learn

The team didn't just guess what would work; they used a rulebook called the Cognitive Theory of Multimedia Learning (CTML).

Think of your brain as having two "loading docks": one for sight (reading text, looking at diagrams) and one for sound (listening).

The Old Way: The original documentation dumps everything into the "Sight" dock. It's so crowded that the dock gets jammed, and your brain can't process the info.
The VisDoc Way: VisDoc splits the work. It puts the text on the "Sight" dock and the explanation in a video/audio on the "Sound" dock. This balances the load, so your brain doesn't get overwhelmed.

The Experiment: Map vs. Manual

To test if VisDoc actually works, the researchers ran a study with 14 new contributors (newcomers). They split them into two groups:

Group A (The Control): They got the original, messy documentation but were allowed to use ChatGPT to help them ask questions.
Group B (The Treatment): They used VisDoc (the interactive map with videos) and nothing else.

The Results:

Success Rate: The VisDoc group was much more successful. They completed nearly all their tasks (20 out of 21), while the ChatGPT group only finished about 60% of theirs (13 out of 21).
Mental Load: The VisDoc group felt much less stressed. They reported significantly lower "mental demand" and "frustration."
- Why? The ChatGPT group often got lost. They had to figure out how to ask the AI the right question, then check if the AI's answer matched the messy manual. It was like trying to navigate a city with a GPS that gives you wrong turns and a map that has no streets drawn on it.
- The VisDoc group just followed the clear path. One participant said, "VisDoc made the steps very clear. I didn't feel lost at any point."
Usability: The VisDoc group rated the tool as much easier to use and learn.

What This Means (According to the Paper)

The paper concludes that restructuring documentation is just as important as writing it.

AI isn't a magic wand: Just giving newcomers access to a chatbot (like ChatGPT) alongside bad documentation doesn't fix the problem. In fact, it can make things worse because the AI and the manual often contradict each other, creating more confusion.
Structure matters: By taking existing information and reorganizing it into a clear, visual, step-by-step workflow (using the rules of how our brains learn), you can drastically reduce frustration and help people succeed.
The Future: The researchers suggest that instead of just asking AI to "write more text," we should use AI to restructure information into visual, task-based guides that match how humans naturally process complex tasks.

In short: If you want to help new people join a project, don't just give them a bigger book. Give them a clear, visual map with a guide who speaks to them. That's what VisDoc does, and the study proves it works better than just having a chatbot help you read a confusing manual.

Technical Summary: Restructuring Onboarding Documents to Reduce Cognitive Overload Using AI

Problem Statement

Open Source Software (OSS) onboarding documentation is frequently characterized as dense, inconsistently structured, and fragmented. While prior research has addressed content gaps using Generative AI (GenAI) to generate additional explanations, the representation issues of existing documentation remain largely unexplored. Newcomers face significant cognitive overload because they must mentally reconstruct multi-step workflows, infer dependencies, and integrate scattered information across multiple files and links. This "text-heavy" presentation imposes excessive working-memory load, leading to confusion, errors, and contributor abandonment. Current interventions, including LLM-based augmentation, often operate within the constraints of existing linear text formats, failing to reorganize the underlying structure of the information.

Methodology

The authors propose VisDoc, a prototype system that leverages GenAI to restructure OSS onboarding documentation based on the Cognitive Theory of Multimedia Learning (CTML). The approach is grounded in the premise that restructuring information to align with human cognitive processing limits can reduce overload.

Theoretical Framework

The study identifies five specific forms of cognitive overload in OSS documentation and maps them to CTML design strategies:

Essential Overload (High Complexity): Addressed via Segmenting (breaking content into task-based units) and Pretraining (providing conceptual overviews).
Extraneous Overload (Confusing Presentation): Addressed via Aligning (grouping related information) and Eliminating (removing redundancy).
Extraneous Overload (Extraneous Materials): Addressed via Signaling (highlighting core steps) and Weeding (removing non-essential details).
Modality Overload (Visual Channel Overuse): Addressed via Off-loading (shifting explanations to the auditory channel via audio/video).
Representational Overload (Working-Memory Burden): Addressed via Synchronizing (pairing text with visuals) and Individualizing (adapting detail levels to prior knowledge).

System Implementation (VisDoc)

VisDoc is a web-based prototype (Python backend, React frontend) that operationalizes these strategies through a GenAI pipeline:

Retrieval & Preprocessing: Fetches Markdown documentation and linked files.
Segmentation & Inference: Uses LangChain's Semantic Chunker refined by GPT-4o to segment documents into task-based units, infer task dependencies, and generate workflow structures. A Retrieval-Augmented Generation (RAG) framework (Cognita) is employed to mitigate hallucinations.
Multimodal Generation: Generates instructional scripts, executes them via Claude Computer Use to create video walkthroughs, and synthesizes narration using ElevenLabs.
Interface: Presents the restructured content as an interactive task tree. Users can expand/collapse nodes, search for specific tasks, and access synchronized text, diagrams, and video walkthroughs.
Human-in-the-Loop: Recognizing the risks of AI hallucination, VisDoc includes an "Edit Tree" feature allowing maintainers to review, refine, and curate the generated content before it is used.

Evaluation Design

The study employed a two-phase evaluation:

Expert Evaluation (N=4): OSS practitioners and researchers evaluated VisDoc's completeness, accuracy, granularity, and adoptability using the Transformers project.
Between-Subjects User Study (N=14): Newcomers were randomly assigned to either the VisDoc group or a Control Group (Original CONTRIBUTING.md + optional ChatGPT assistance). Participants completed three onboarding tasks (creating a PR, translating documentation, creating a script).
- Metrics: Task success (binary), Cognitive Load (NASA-TLX), and Usability (System Usability Scale - SUS).
- Analysis: Quantitative analysis using Fisher's Exact Test and Mann-Whitney U tests; qualitative analysis of think-aloud protocols.

Key Results

Expert Evaluation

Experts affirmed that VisDoc captured all essential information, preserved the original intent and ordering of tasks without hallucination, and provided an effective balance between high-level overviews and detailed guidance. They viewed the tool as feasible for integration into real OSS workflows.

User Study Findings

Task Performance: The VisDoc group achieved a significantly higher success rate (20/21 tasks) compared to the Documentation+ChatGPT group (13/21 tasks) ( $p=0.02$ , effect size = 0.89). Participants using VisDoc reported that the structured layout reduced uncertainty and prevented common errors, whereas the control group struggled with prompt engineering and conflicting information.
Cognitive Load: VisDoc users reported significantly lower cognitive load across four of five NASA-TLX dimensions:
- Mental Demand: Median 30 (VisDoc) vs. 75 (Control) ( $p=0.04$ ).
- Temporal Demand: Median 20 vs. 60 ( $p=0.04$ ).
- Performance: Median 0 vs. 30 ( $p=0.009$ ).
- Frustration: Median 10 vs. 60 ( $p=0.004$ ).
- Note: Effort showed a lower median for VisDoc but was not statistically significant ( $p=0.48$ ).
Usability: VisDoc received significantly higher SUS scores than the control group ( $p=0.005$ , Cohen's $d=0.77$ ), with users describing the interface as intuitive, coherent, and less distracting.

Significance and Contributions

The paper claims four primary contributions:

Theory-Grounded Analysis: A characterization of five specific forms of cognitive overload in OSS onboarding and the derivation of evidence-based design strategies to mitigate them using CTML.
GenAI Pipeline: A novel pipeline that operationalizes cognitive design principles to automatically segment, reorganize, and generate multimodal instructional materials from raw documentation.
System Implementation: The design and open-source release of VisDoc, a system that transforms linear documentation into an interactive, multimodal task tree.
Empirical Validation: Evidence demonstrating that cognitively informed restructuring significantly improves task success, reduces cognitive load, and enhances usability compared to traditional documentation supplemented by LLMs.

The authors conclude that while LLMs are effective micro-level assistants, they often fail to provide the macro-level structural guidance necessary for complex workflows. By combining AI's generative capabilities with human-centered cognitive theory and human oversight, VisDoc demonstrates a path toward more trustworthy and effective onboarding tools that reduce the cognitive barriers to OSS participation.

Limitations

The authors acknowledge several limitations:

Sample Size: The user study involved a small, predominantly male, university-based sample (N=14), limiting generalizability to the broader OSS population.
Single-Project Scope: Evaluation was conducted on a single project (Transformers) with controlled tasks, which may not reflect the diversity of OSS workflows or the longitudinal nature of real onboarding.
Dependency on Source Quality: VisDoc reorganizes existing content but does not generate new information; poor source documentation leads to suboptimal outputs.
Hallucination Risks: Despite RAG and human-in-the-loop design, AI inaccuracies remain a risk requiring maintainer vigilance.
Resource Intensity: The pipeline requires significant computational resources and API costs, which may hinder adoption for smaller projects.
Short-Term Focus: The study measured immediate task performance and perceptions, not long-term contributor retention or engagement.