📊 statistics

Automated Residual Plot Assessment With the R Package autovi and the Shiny Application autovi.web

This paper introduces the R package `autovi` and its Shiny application `autovi.web`, which utilize a computer vision model to automate the assessment of residual plots by predicting visual signal strength, thereby addressing the scalability and consistency limitations of manual visual diagnosis in linear modeling.

Original authors: Weihao Li, Dianne Cook, Emi Tanaka, Susan VanderPlas, Klaus Ackermann

Published 2026-06-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Weihao Li, Dianne Cook, Emi Tanaka, Susan VanderPlas, Klaus Ackermann

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Problem: The "Human Eye" Bottleneck

Imagine you are a detective trying to solve a crime. You have a suspect (a statistical model) and a pile of evidence (data). To see if your suspect is guilty, you look at the "residuals"—the leftover clues that the suspect couldn't explain.

In statistics, analysts usually look at these clues by drawing a picture (a residual plot) and using their eyes to see if anything looks "weird."

The Issue: Humans are bad at this. We get tired, we see patterns that aren't there (like seeing a face in a cloud), or we miss patterns that are actually there. It's like trying to find a specific needle in a haystack by squinting at it for hours.
The Old Fix: Sometimes, statisticians use a "lineup" protocol. They show you the real picture mixed in with 19 fake pictures. If you can spot the real one, it means there's a problem. But this is slow and still relies on human eyes.

The Solution: A Robot Detective

The authors of this paper built a new tool called autovi (Automated Visual Inference). Think of it as a robot detective that never gets tired and has superhuman eyesight.

Instead of asking a human to squint at a graph, this tool uses Computer Vision (the same technology that lets self-driving cars "see" the road) to look at the residual plots.

How the Robot "Sees"

Training: The robot was trained on thousands of fake scenarios. It was shown pictures of "perfect" models (where the clues are scattered randomly) and "broken" models (where the clues form a clear pattern, like a curve or a funnel).
The Score (VSS): When you give the robot a new picture, it doesn't just say "yes" or "no." It gives it a score called Visual Signal Strength (VSS).
- Low Score: "I see nothing suspicious. The clues look random. Your model is probably fine."
- High Score: "I see a clear pattern! The clues are lined up in a way they shouldn't be. Your model is missing something."

The Two Tools: The Toolbox and the Web App

The paper introduces two ways to use this robot:

1. The R Package (autovi): The Professional Toolbox
This is for people who are comfortable with coding (specifically in R).

How it works: You feed your data into the package. It automatically generates the plots, runs them through the robot, and gives you a report.
The Report: It tells you the VSS score and a "p-value" (a probability number). If the number is low, it means the robot is very confident that your model has a problem.
Visuals: It can show you a "lineup" (the real plot mixed with fake ones) so you can double-check the robot's work, or a density chart that shows how your data compares to the "perfect" random data.

2. The Web App (autovi.web): The User-Friendly Kiosk
Not everyone knows how to code. For them, the authors built a website.

No Installation: You don't need to install complex software or manage different programming languages. You just open a browser.
The Magic: The website runs the heavy computer vision work right in your browser (using a technology called TensorFlow.js). It's like having a supercomputer in your web browser that doesn't need to be plugged into a server farm.
Ease of Use: You upload your data (or use their examples), click a button, and it instantly shows you the robot's assessment, the lineup of plots, and even a "heat map" showing exactly where in the picture the robot saw the problem.

A Real-World Example from the Paper

The authors tested their tool on three scenarios:

The False Alarm: A plot looked messy (like a funnel), which usually means a broken model. But the robot said, "Actually, this is fine. It just looks messy because you have fewer data points on one side." The robot saved the analyst from panicking.
The Hidden Curve: A plot looked mostly okay, but there was a tiny curve hidden in the noise. A standard computer test missed it, but the robot spotted it immediately.
The Obvious Pattern: A plot had a weird shape hidden inside it (inspired by a quote from a famous statistician). The robot gave it a very high "danger" score, correctly identifying the issue.

The Bottom Line

This paper presents a new way to check if your statistical models are working correctly.

Before: You had to squint at graphs and hope you didn't miss anything.
Now: You have a robot that scans the graph, compares it to thousands of "perfect" examples, and gives you a clear score and a probability that something is wrong.

It makes the process faster, more consistent (the robot doesn't get tired), and accessible to people who aren't coding experts, thanks to the free web app.

Technical Summary: Automated Residual Plot Assessment with `autovi`

Problem Statement
Visual assessment of residual plots is a standard approach for diagnosing linear models, yet it suffers from significant limitations. Manual evaluation does not scale well and often leads to inconsistent decisions across different analysts. While the "lineup protocol"—embedding an observed plot among null plots—reduces subjectivity, it requires substantial human effort, making it impractical for large-scale applications. Existing software packages (e.g., lmtest, DHARMa, ggResidpanel) provide conventional tests and graphical diagnostics, but reliance on subjective interpretation of these plots can lead to issues such as over-interpreting random patterns as model violations. There is a need for an automated, scalable method to assess residual plots that maintains the rigor of visual inference without the burden of manual labor.

Methodology
The paper introduces a framework that automates residual plot assessment using computer vision and statistical testing. The core of this methodology involves the following components:

Visual Signal Strength (VSS): A continuous metric is defined to quantify the clarity of visual patterns in a diagnostic plot that may indicate model violations. Theoretically, VSS estimates the distance between the residual distribution of a fitted model and a reference distribution (expected under correct model specification). This distance is based on the Kullback-Leibler (KL) divergence. Since the true residual distribution is unknown in practice, the VSS is approximated by a computer vision model trained on synthetic data.
Computer Vision Model: A convolutional neural network (CNN) is trained on a large dataset of synthetic regression models. These models simulate specific violations of classical linear regression assumptions, including non-linearity (via Hermite polynomial transformations), heteroscedasticity (error variance dependent on predictors), and non-normality (discrete, uniform, and lognormal distributions). The model takes a residual plot image and auxiliary variables (e.g., number of observations) as input and outputs a predicted VSS.
Statistical Testing Protocol:
- Null Residuals: Generated under the null hypothesis that the model is correctly specified (typically by sampling from a normal distribution with mean zero and estimated error variance).
- Bootstrapped Residuals: Generated by refitting the model on bootstrap samples to estimate the variation in VSS due to sampling variability.
- Hypothesis Testing: The automated assessment computes the proportion of null plots whose VSS equals or exceeds that of the observed residual plot. This proportion serves as a $p$ -value for a one-sided hypothesis test.
Software Implementation:
- R Package (autovi): Built on the bandicoot object-oriented programming system, the package integrates R and Python via the reticulate package. It uses ggplot2 for plot generation, cassowaryr for scagnostics (scatter plot diagnostics), pillow for image processing, and TensorFlow for the pre-trained CNN. The infrastructure is modular, allowing users to override specific steps (e.g., data extraction, residual simulation) to extend the tool to other model classes like Generalized Linear Models (GLMs).
- Shiny Application (autovi.web): A web interface that allows users to perform assessments without local installation of R or Python dependencies. It utilizes TensorFlow.js to execute the computer vision model directly in the browser, shifting computational load to the client side.

Key Contributions

Automation of Visual Inference: The paper presents the first automated implementation of the lineup protocol for residual plots, replacing manual visual comparison with a computer vision-based statistical test.
Quantitative Metric (VSS): It introduces a continuous, interpretable metric (VSS) that quantifies the strength of visual patterns, bridging the gap between subjective visual assessment and objective statistical testing.
Modular Framework: The autovi package is designed with a modular architecture that supports extension beyond simple linear regression. The authors demonstrate this by creating a custom checker for Poisson regression, showing how users can override specific modules (e.g., null residual simulation) to accommodate different model classes.
Accessibility: By providing both an R package and a browser-based Shiny application, the authors lower the barrier to entry for automated diagnostics, making advanced visual inference accessible to users without programming expertise or complex environment setup.

Results and Illustrative Examples
The paper demonstrates the tool's efficacy through several examples:

False Positives: In a simulated case where apparent heteroscedasticity was caused by skewness in predictor variables rather than a true model violation, the automated tool correctly assigned a low VSS (1.53) and a non-significant $p$ -value (0.25), whereas a human might misinterpret the pattern.
Non-linearity Detection: Using the mtcars dataset, the tool detected a subtle non-linear structure (VSS = 3.57, $p < 0.05$ ) that was confirmed by a Ramsey RESET test.
Complex Patterns: In a "surreal" dataset with embedded structure, the tool identified a strong violation (VSS = 5.87, $p < 0.05$ ) that standard non-linearity tests (RESET) missed, though other tests (Breusch-Pagan, Shapiro-Wilk) detected related issues.
Extension to GLMs: A Poisson regression example with an omitted quadratic term resulted in a U-shaped pattern in the lineup. The custom checker successfully identified this with a VSS of 4.875 and a $p$ -value of 0.0099, demonstrating the framework's adaptability.

Significance and Claims
The authors claim that autovi and autovi.web address a critical gap in statistical software by automating a task that has historically relied on inconsistent human interpretation. The significance of this work lies in:

Efficiency and Consistency: Automating the assessment improves the efficiency of the diagnostic process and promotes consistency across different users and studies.
Scalability: The approach overcomes the labor constraints of the lineup protocol, making visual inference feasible for large-scale applications.
Accessibility: The web interface removes technical barriers (installation, dependencies), allowing a broader audience, including those without programming experience, to utilize advanced diagnostic tools.
Foundation for Future Work: The modular framework establishes a foundation for extending automated assessment to other diagnostic plots and statistical models, potentially revolutionizing how visual displays are used in statistical inference.

The paper concludes that these tools allow researchers to focus more on interpreting results and refining models rather than grappling with the intricacies of manual plot assessment, while maintaining the rigor of visual inference.