Technical Summary: Scaling Participation in Modular AI Systems

Problem Statement

Current Large Language Models (LLMs) are predominantly developed by a small number of private firms as centralized, monolithic entities. This "top-down" approach creates a structural mismatch between the uniform nature of these models and the multifaceted diversity of human knowledge, reasoning, and values. The opacity of these black-box systems erodes public trust, while their training on skewed data and benchmarks prevents them from credibly representing the pluralistic needs of society. Furthermore, the reliance on single, massive models limits the ability to incorporate diverse, specialized priorities from different stakeholders. The paper argues that for AI to mediate decisions affecting all of humanity, it must transition from centralized silos to an open, bottom-up, and collaborative paradigm.

Methodology

The authors propose Scaling Participation, a paradigm where modular AI systems are constructed from the bottom up through the contributions of diverse stakeholders. The methodology consists of two primary phases:

1. Soliciting Participation

The authors curated a pool of 61 language models released as academic research artifacts. These models were contributed by researchers from 236 academic labs globally, representing diverse architectures, training domains, and research agendas beyond commercial interests. These models serve as specialized, modular building blocks rather than attempts to create a single unified foundation model.

2. Orchestrating Participation

The study employs 14 distinct collaborative algorithms across three levels of information exchange to integrate these diverse models into compositional systems:

• API-level Collaboration: Focuses on routing and selection. Methods include Prompt Routing, Trained Router, Graph Routing, and Switch Generation, where a controller selects or switches between candidate models based on input.
• Text-level Collaboration: Involves the exchange of generated text. Methods include Multi-Agent Refine (iterative debate and refinement), Multi-Agent Finetuning, LLM Blender (ranking and fusing responses), AggLM (reinforcement learning for aggregation), Sparta Alignment (competitive reputation-based alignment), and Heterogeneous Swarms (DAG-based orchestration).
• Weight-level Collaboration: Involves arithmetic operations on model parameters. Methods include Dare-Ties (pruning and sign consensus), Greedy Soup (iterative weight averaging), Weight Extrapolation (ExPO), and Model Swarms (particle swarm optimization in parameter space).
  • Note: For weight-level methods requiring identical architectures, the authors performed knowledge distillation from the diverse contributed models into a unified student architecture (Qwen-2.5-7B) to preserve specialization while enabling parameter merging.

The experiments scaled the number of participating models from 2 up to 32 within these collaborative frameworks, testing them against non-modular and non-participatory baselines.

Key Results

Extensive evaluations across 5 domains (General QA, Reasoning, Knowledge, Safety, Instruction Following) and 15 tasks yielded the following findings:

• Performance Superiority: Participatory AI systems outperformed non-modular and/or non-participatory baselines by up to 15.4% on average. Notably, the best collaborative systems outperformed the largest monolithic baseline (Llama 3.1 405B), which is 2.38 times larger than the combined size of all 32 participating models.
• Scaling Trend: Performance consistently improved as the number of participating models increased from 2 to 32, with an average improvement of 28.78% across the scaling range.
• Diversity as a Driver: Controlled experiments demonstrated that improvements were not merely due to increased compute but were significantly driven by model diversity. Systems built with models from diverse institutions and international sources showed greater performance gains (up to 18.8% and 18.0% respectively) compared to systems with less diverse compositions.
• Collaborative Emergence: The systems exhibited emergent capabilities, solving over 15% of problems where all individual participating models failed. This suggests that collaboration allows the system to generalize beyond the capabilities of its individual components.
• Representation of Human Diversity: Participatory systems showed a unique advantage in representing human diversity, outperforming monolithic models by 21.58% on tasks involving cultural contexts and values (CultureBench, Value Kaleidoscope), a higher margin than on general utility tasks.
• Compositional vs. Individual Strength: The study found no correlation between a model's individual strength and its "compositional strength" (its contribution to a collaborative system). This indicates that optimizing for individual performance does not guarantee utility in a participatory framework.
• Participant Benefit: Evaluations on the original priorities of model contributors confirmed that the compositional systems improved performance on the specific tasks and domains the contributors originally targeted, providing a tangible incentive for participation.

Significance and Claims

The paper claims to provide a technical foundation for transitioning AI development from a monolithic status quo to an open, democratized, and collaborative future. Its significance lies in several areas:

1. Paradigm Shift: It demonstrates that "bottom-up" AI, built from diverse, specialized contributions, can rival and surpass centralized, state-of-the-art monolithic models.
2. Pluralistic Alignment: It offers a mechanism to reflect the richness of human talent and needs in AI systems, addressing the democratic legitimacy gap in current AI governance.
3. New Evaluation Metric: It introduces the concept of "compositional strength," arguing that future AI research must evaluate models based on their utility within collaborative systems rather than solely on individual benchmarks.
4. Feasibility: By leveraging existing open-source models and requiring only consent for use (rather than workflow changes), the approach is presented as a low-barrier, immediately actionable path toward participatory AI.

The authors conclude that scaling participation is not just a technical optimization but a necessary evolution to build AI systems that are more transparent, pluralistic, and capable of serving the diverse needs of society.