Technical Summary: Assessing the Variety of a Concept Space Using an Unbiased Estimate of Rao's Quadratic Index

1. Problem Statement

The paper addresses the challenge of quantifying "variety" (or diversity) within a design concept space during the early conceptual design phase. While past research links design creativity to divergent thinking and the breadth of concept exploration, existing metrics for assessing variety suffer from significant limitations:

• Reliance on Genealogy Trees: Most existing metrics (e.g., Shah et al., Nelson et al., Verhaegen et al., Ahmed et al.) require a tree-based representation where concepts are classified into discrete nodes at predefined abstraction levels (physical principle, working principle, embodiment, detail). This binary approach fails to capture real-valued distances (e.g., semantic similarity) between ideas, treating all distinct nodes as equally distant.
• Lack of Accuracy and Sensitivity: The authors identify two critical flaws in existing metrics through theoretical test cases:
  • Accuracy: Some metrics violate the mathematical definition of variety (the probability that two randomly selected concepts belong to different groups). Specifically, the Herfindahl-Hirschman Index for Design (HHID) uses a biased estimator that yields inaccurate scores for small sample sizes.
  • Sensitivity: Metrics like those by Shah and Nelson fail to reflect changes in the distribution of concepts across nodes (e.g., they produce identical scores for even and skewed distributions unless the number of nodes changes).
• Ambiguity in Abstraction: Defining clear boundaries between abstraction levels (e.g., physical vs. working principles) is often difficult for designers, leading to inconsistent tree constructions.
• Lack of Automation: No existing software tools automate variety assessment directly from textual concept data.

2. Methodology

The authors propose a new prescriptive framework and a software tool, VariAnT (Variety Assessment Tool), to overcome these limitations. The methodology consists of three main components:

A. Flexible Knowledge Representation

Instead of forcing concepts into a rigid genealogy tree, the framework allows concepts to be represented using any preferred abstraction scheme. The authors utilize the SAPPhIRE (States, Actions, Parts, Phenomena, Inputs, oRgans, Effects) causality model as a demonstration, which supports multi-state systems and complex causal chains. Concepts are stored in a structured data frame where each concept consists of instances corresponding to different abstraction levels.

B. Vector Encoding and Distance Calculation

To measure the "distance" between concepts without relying on discrete nodes:

1. Textual Aggregation: Descriptions of a concept at a specific abstraction level are concatenated into a single string.
2. Vector Embedding: These strings are converted into numerical vectors using Natural Language Processing (NLP) techniques, specifically Sentence-BERT (S-BERT) (e.g., the all-MiniLM-L6-v2 model).
3. Distance Matrix: A pairwise distance matrix is generated using the cosine distance ($d = 1 - \text{cosine similarity}$) between the vector embeddings of every pair of concepts at each abstraction level. This allows for continuous, real-valued distance measurements rather than binary distinctions.

C. The New Metric: Rao's Quadratic Index for Design (RQID)

The core of the new metric is an unbiased estimate of Rao's Quadratic Diversity Index.

• The variety score for a concept space at a specific abstraction level is calculated as the average pairwise distance between all concepts.
• Mathematically, this is equivalent to an unbiased estimator of the Gini-Simpson Index (GSID) when distances are binary, but it extends to continuous distances.
• The final variety score is a weighted average of scores across all abstraction levels, where weights ($w_\alpha$) can be assigned by the user to reflect the relative importance of specific levels.

3. Key Contributions

1. New Variety Metric (RQID): The paper introduces a distance-based metric that eliminates the need for genealogy trees. It uses real-valued distances derived from semantic similarity, offering greater flexibility and accuracy.
2. Theoretical Validation: Through two test cases, the authors demonstrate that existing metrics lack sensitivity to distribution skewness and accuracy in small sample sizes. They show that the proposed RQID (an unbiased estimator) satisfies the mathematical definition of variety more robustly than previous methods.
3. Software Tool (VariAnT): The authors developed a Python-based GUI tool that automates the entire assessment process, from importing textual concept data to generating variety scores, individual concept scores, heatmaps, and clustering dendrograms.
4. Framework Generalizability: The proposed framework is not limited to SAPPhIRE; it is designed to accommodate any representation scheme and any distance-assessment method, allowing designers to tailor abstraction levels to their specific context.

4. Results and Evaluation

The framework was evaluated through two primary examples:

• Synthetic Example (Boiling Water): Four concepts generated by an AI (Electric Kettle, Gas Stove, Solar Heater, Friction Heater) were analyzed using SAPPhIRE levels. The tool successfully calculated an overall variety score ($V=0.378$) and identified that the "Friction Heater" was the most distinct concept. The analysis also revealed that variety varied significantly across abstraction levels (e.g., higher diversity at the "State" level for the Friction Heater), providing actionable insights for designers to diversify specific idea spaces.
• Comparative Evaluation (Snowy Transportation): The tool was tested on a dataset of 23 concepts for transportation in snow (adapted from Henderson et al., 2017).
  • Differentiation: The dataset was split into two sets (P and Q) that produced identical genealogy trees under existing methods, resulting in identical variety scores.
  • Sensitivity: The proposed framework detected that Set Q was conceptually more heterogeneous at the physical principle level. Consequently, it assigned a higher variety score to Set Q ($0.692$) compared to Set P ($0.627$), demonstrating the metric's ability to capture conceptual distance that tree-based methods miss.

5. Significance and Claims

The paper claims that the proposed framework offers a more robust and flexible approach to assessing design variety by:

• Decoupling from Tree Structures: It removes the rigid requirement for discrete, hierarchical classification, allowing for the use of continuous semantic distances.
• Improving Accuracy: By using an unbiased estimator (RQID), it corrects the statistical biases found in previous metrics like HHID, particularly for smaller concept sets.
• Enhancing Sensitivity: It accurately reflects changes in the distribution of concepts, distinguishing between mere "fluency" (quantity) and true "variety" (diversity of underlying principles).
• Practical Utility: The tool provides visualizations (heatmaps, dendrograms) that help designers identify clusters of similar ideas and pinpoint abstraction levels where diversity is low, thereby guiding further exploration.

The authors maintain a modest stance, acknowledging limitations such as the computational complexity ($O(N^2)$), the reliance on the quality of textual descriptions, and the current lack of integration with image or CAD data. They suggest future work should focus on optimizing abstraction level weights, integrating alternative embedding models, and applying the tool to real-world design-by-analogy scenarios.