PriorWeaver: Prior Elicitation via Iterative Dataset Construction

PriorWeaver is an interactive visualization system that simplifies Bayesian prior elicitation for novice analysts by allowing them to iteratively construct and refine datasets reflecting their beliefs, resulting in priors that are more aligned with their expectations compared to traditional methods.

The Gist

Imagine you are a chef trying to teach a robot how to cook your famous soup.

In the old way of doing things (traditional statistics), you would have to speak the robot's language: "Add exactly 3.14 grams of salt with a variance of 0.5." You'd have to translate your gut feeling ("It needs to be salty, but not too salty") into complex math formulas. If you get the math wrong, the robot makes a disaster, and you have no idea why.

PriorWeaver is a new tool that changes the game. Instead of speaking math, it lets you speak reality.

Here is how it works, using simple analogies:

1. The Problem: The "Translator" Gap

In Bayesian analysis (a way of learning from data), you need to tell the computer what you already believe before you start looking at new data. This is called "eliciting a prior."

• The Old Way: You have to guess abstract numbers (like "the slope of the line should be 0.7"). It's like trying to describe a color to someone who only understands hex codes. Most people aren't fluent in "hex code," so they guess, get it wrong, and the robot makes weird soup.
• The New Way (PriorWeaver): You don't guess numbers. You build a fake dataset that looks exactly like the world you know.

2. The Solution: Building a "Dream World"

PriorWeaver treats the process like building a Lego city or a cast of characters for a movie.

• Step 1: The Ingredients (The Histograms)
  Imagine you are thinking about "Age" and "Income." Instead of typing "Average age is 40," you click on a chart to add dots. You add a dot for a 25-year-old, a dot for a 60-year-old, and a bunch of dots for 40-year-olds. You are literally drawing the shape of the population in your head.

  • Analogy: You are filling a bucket with marbles of different sizes to represent the people you know.

• Step 2: The Connections (The Scatterplots & Parallel Coordinates)
  Now, you need to connect the dots. You know that older people usually have more education, but maybe not the highest income.

  • In PriorWeaver, you can "brush" (select) a group of dots. You can say, "Take these 50-year-olds with high education and give them a moderate income."
  • Analogy: It's like casting actors for a play. You pick a 50-year-old actor, give them a script that says "High Education," and assign them a salary of $60k. You do this for hundreds of imaginary people until your "cast" perfectly matches your real-world experience.

3. The Magic: The "Reality Check"

Once you have built your "Dream World" (your dataset of imaginary people), you hit a button called "Translate."

• The Translation: The computer takes your imaginary people, runs the math behind the scenes, and says, "Okay, based on your cast, here is what the future data should look like."
• The Feedback: It shows you a graph of the results.
  • Scenario: You built a world where everyone is rich. The computer says, "Your model predicts everyone will make $1 million."
  • Your Reaction: "Whoa, that's too high! I didn't mean everyone to be rich."
  • The Fix: You go back to your "Dream World," delete a few of the super-rich imaginary people, and add a few average earners. You hit "Translate" again.
  • Result: Now the prediction looks realistic.

Why This Matters

The paper tested this with 17 people who knew statistics but had never used Bayesian methods before.

• Without PriorWeaver: They felt like they were guessing in the dark. They had to do mental math to convert "I think rich people are rare" into "The parameter must be 0.04." They felt confused and unsure.
• With PriorWeaver: They felt like directors. They could see their beliefs, touch them, and tweak them.
  • They said it felt "intuitive" because they were just adding examples they knew from real life.
  • They felt confident because they could see exactly why the computer was making a weird prediction (e.g., "Oh, I added too many high-income 20-year-olds").

The Big Takeaway

PriorWeaver turns a scary math problem into a creative design task.

Instead of asking you to be a mathematician, it asks you to be a storyteller. It says, "Don't tell me the numbers. Just show me the story of the world you believe in." Once you've told the story, the computer handles the heavy lifting of the math.

This makes a complex scientific method accessible to anyone who can imagine a scenario, opening the door for more people to use powerful tools to understand the world.

Technical Summary

Here is a detailed technical summary of the paper "PriorWeaver: Prior Elicitation via Iterative Dataset Construction."

1. Problem Statement

Prior elicitation is the critical yet challenging step in Bayesian analysis where analysts translate their domain knowledge and beliefs into probability distributions (priors) for model parameters.

• The Core Challenge: Existing tools typically operate in parameter space, requiring users to define abstract statistical parameters (e.g., regression coefficients, variance) using mathematical distributions. This creates a high cognitive barrier, especially for domain experts who lack formal Bayesian training.
• Limitations of Current Tools:
  • They require translating real-world concepts (e.g., "older people earn more") into abstract parameters (e.g., "coefficient $\beta \sim Normal(3, 0.5)$").
  • They often fail to capture relational knowledge between variables.
  • They provide limited feedback, forcing users to rely on trial-and-error to refine priors.
  • They are often designed for experts, not novices, leading to a dependency on "facilitators" (Bayesian experts) to set priors.

2. Methodology: PriorWeaver

PriorWeaver reframes prior elicitation as an iterative dataset construction problem. Instead of asking users to define parameters directly, it allows them to construct a synthetic dataset that represents their beliefs about observable variables. The system then derives statistical priors from this dataset.

A. System Architecture & Workflow

The system operates through a four-step iterative loop:

1. Externalize (Observable Space): Analysts express beliefs about individual variables and their relationships using coordinated interactive visualizations.
2. Construct (Dataset Generation): The system builds a concrete dataset based on these inputs.
3. Derive (Statistical Priors): The system fits a predefined statistical model (Generalized Linear Models) to the constructed dataset using bootstrapping to generate parameter estimates and fit prior distributions.
4. Evaluate (Prior Predictive Checks): The system generates predictive distributions of the outcome variable based on the derived priors. Analysts compare these predictions against their original assumptions to identify discrepancies and refine the dataset.

B. Key Technical Components

• Coordinated Visualizations:
  • Univariate Histograms: Users add/remove data points to define marginal distributions (frequency-based reasoning rather than probability allocation).
  • Bivariate Scatterplots: Users brush regions to define pairwise relationships and use a "Generate" function to create synthetic examples within those regions.
  • Parallel Coordinates Plot: Users define multivariate relationships by brushing across multiple axes. This view handles high-dimensional constraints and allows users to connect incomplete data entries (rows with missing values) into complete synthetic cases.
• Dataset Construction Logic:
  • The system manages "Incomplete" (partial variable specification) and "Complete" (full row) entities.
  • A "Connect" function automatically identifies compatible incomplete entries and merges them into complete synthetic cases, ensuring the dataset reflects complex multivariate assumptions.
• Prior Derivation Algorithm:
  1. Filtering: Incomplete rows are removed.
  2. Bootstrapping: 100 datasets are generated by resampling 50 rows with replacement from the constructed dataset to capture uncertainty.
  3. Model Fitting: The statistical model is fit to each bootstrapped dataset to obtain parameter estimates.
  4. Distribution Fitting: Maximum Likelihood Estimation (MLE) is used to fit continuous probability distributions (e.g., Normal, Gamma, Student-t) to the aggregated parameter estimates, forming the final priors.
• Feedback Mechanism (PPCs):
  • The system samples predictor values from the analyst's constructed dataset and combines them with parameters sampled from the derived priors.
  • It visualizes the resulting predictive distribution of the outcome variable, allowing direct comparison with the analyst's expected distribution (e.g., checking for implausible negative incomes).

3. Key Contributions

1. Conceptual Shift: Proposes viewing prior elicitation as dataset construction in the observable space, moving away from abstract parameter specification.
2. PriorWeaver System: An interactive tool that supports iterative dataset construction via coordinated visualizations, derives priors via bootstrapping, and provides actionable feedback through Prior Predictive Checks (PPCs).
3. Design Considerations: Articulates four design principles for belief elicitation:
   • Support expression in the observable space.
   • Support both distributional and relational knowledge.
   • Provide actionable feedback for iterative refinement.
   • Use visual representations (frequency-based) to facilitate reasoning.

4. Results (User Study)

A controlled within-subjects study was conducted with 17 participants who were experienced in statistical modeling (Frequentist) but new to Bayesian analysis. They compared PriorWeaver against a baseline parameter-space tool (using the trial-roulette method).

• Knowledge Externalization (RQ1):
  • Strategies: PriorWeaver users adopted flexible strategies (distribution-driven, association-driven, example-driven).
  • Cognitive Load: Baseline users struggled with "mental translation" and trial-and-error. PriorWeaver users reported significantly higher comfort, clarity, and ease of expression ($p < 0.01$).
  • Relationships: PriorWeaver effectively helped users express complex multivariate relationships, which were difficult to capture in the baseline.
• Evaluation and Refinement (RQ2):
  • Actionability: In the baseline, refinement was frustrating and unpredictable (users didn't know how to adjust parameters to fix predictive errors).
  • Precision: In PriorWeaver, users could directly target discrepancies (e.g., "add a data point for high education/low income" to fix a tail).
  • Outcome: PriorWeaver users produced priors that were better aligned with their expectations from the first iteration and achieved more reasonable final predictive distributions than the baseline.
• Attitudes (RQ3):
  • Participants felt PriorWeaver made Bayesian analysis approachable and transformed it from a theoretical concept to a practical tool.
  • They reported higher confidence and a significantly increased willingness to use Bayesian methods in future research.

5. Significance and Implications

• Lowering Barriers: PriorWeaver significantly reduces the expertise required to perform prior elicitation, making Bayesian analysis accessible to domain experts without formal training.
• Cognitive Alignment: By grounding the process in concrete, observable data (frequency formats) rather than abstract parameters, the system aligns with natural human reasoning and mental models.
• Iterative Sensemaking: The system transforms prior elicitation from a one-off specification task into a sensemaking loop, where users can visualize the consequences of their assumptions and refine them purposefully.
• Future Directions: The authors suggest extending the system to support categorical variables, non-linear models, and more complex variable interactions, as well as integrating AI to help validate the "veracity" of expressed beliefs.

In summary, PriorWeaver demonstrates that shifting the abstraction level of prior elicitation from parameters to constructible datasets effectively bridges the gap between domain knowledge and statistical modeling, leading to more accurate, interpretable, and user-aligned Bayesian analyses.