📊 statistics

Attractor strength identifies classifier label geometry as construct- irrelevant variance in machine-scored developmental drawing assessment

This study demonstrates that in machine-scored developmental drawing assessments, the perceived difficulty of drawing specific objects is largely driven by construct-irrelevant variance stemming from the scoring pipeline's classifier label geometry rather than the child's actual developmental ability, thereby rendering single-prompt scores unreliable and necessitating attractor-adjusted metrics for valid cross-pipeline comparisons.

Original authors: Rafat Saleh Madani

Published 2026-10-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Rafat Saleh Madani

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

For a hundred years, psychologists have believed that a child's drawing is a direct window into their mind. The idea is simple and enduring: if you ask a child to draw a person or a house, the quality of that sketch reveals how their brain has developed. This belief has fueled decades of testing, where educators and clinicians look at a child's artwork to gauge their intelligence or emotional health. The assumption has always been that the drawing is a neutral reflection of the child, and that the specific object they were asked to draw—a cat, a tree, or a car—does not matter. The object is just a prompt, a blank slate upon which the child's abilities are projected. But what if the object itself is not neutral? What if the specific thing a child is asked to draw changes the score more than the child's own development does?

A new study challenges this century-old assumption by looking at thousands of drawings through the lens of modern computer vision. Researchers gathered a massive collection of 37,770 drawings made by over 8,000 children between the ages of two and ten. These children were asked to draw 48 different objects, ranging from animals like cats and whales to everyday items like chairs and scissors. Instead of having human experts grade the drawings, the researchers used two different computer programs to score them. These programs, trained to recognize sketches, assigned a score based on how well the drawing matched the intended object. The goal was to see if the score truly reflected the child's skill or if it was secretly being driven by the difficulty of the object itself.

The results were startling. When the researchers broke down the numbers, they found that the object the child was asked to draw mattered more than the child who drew it. In fact, the specific category of the object accounted for more of the variation in scores than the child's age or ability did. To put this in perspective, the difference in difficulty between drawing a "snail" and drawing a "tiger" was equivalent to a difference of four to seven years of a child's natural development. If a child drew a snail, they received a score that looked like they were much older and more skilled than a child who drew a tiger, even if both children were the same age and had the same ability. The computer programs were not just measuring the child; they were heavily influenced by the specific prompt.

The researchers dug deeper to understand why this happened. They checked if the results were simply because some objects are harder to draw than others, or because children practice drawing some things more often. They adjusted the scores to account for how well a child could trace shapes, how long they spent drawing, and how familiar the object was to them. Even after making these adjustments, the object still dominated the score. The study ruled out the idea that this was just a matter of motor skills or practice. Instead, the problem lay in how the computer programs themselves worked.

The computer programs used to score the drawings have a hidden bias built into their design. They do not treat all objects as equally likely to appear. Some objects, like a snail, are "attractors" for the computer; the program is naturally inclined to see a snail even when looking at a drawing of something else. Other objects, like a tiger, are harder for the program to recognize, so the program rarely assigns that label unless the drawing is perfect. This means a child drawing a snail gets a high score simply because the computer is eager to say "snail," while a child drawing a tiger gets a low score because the computer is hesitant. This bias is a property of the computer's internal logic, not the child's ability. It is a form of unfairness that has never been seen before in psychological testing, where the scoring tool itself creates the difficulty.

The consequences of this finding are significant for how we assess children. The study showed that using a single drawing to judge a child's development is unreliable. If a child is asked to draw one object, their score is likely to be more about that object's difficulty than their actual mind. To get a reliable score that could be used for individual decisions, a child would need to draw between 26 and 32 different objects. This is far more than any current test asks for. The researchers also found that the difficulty of an object is not a fixed fact. A drawing of a snail might be easy for one computer program but hard for another, depending on how that specific program was built. This means that "norm tables," which are lists used to compare a child's score against others, cannot be trusted to work across different testing systems.

Ultimately, the study reveals that the window we thought we were looking through is actually a frame that distorts the view. The score a child receives is a mix of three things: the child's ability, the object they drew, and the computer's bias. For the first time, the researchers have shown how to separate the computer's bias from the child's ability. By measuring how much a computer program is naturally drawn to certain labels, they can adjust the scores to remove this artificial difficulty. This adjustment does not change the child's score, but it changes what that score means. It allows researchers to see the child's true development without the interference of the computer's preferences. The lesson is clear: to understand a child's mind through their art, we must first understand the tools we use to measure it.

Technical Summary: Attractor Strength Identifies Classifier Label Geometry as Construct-Irrelevant Variance

1. Problem Statement

The paper addresses a fundamental validity threat in machine-scored developmental drawing assessments: the assumption that a drawing score reflects the child's cognitive development rather than the specific object (stimulus) the child was asked to draw. While classical and computational literatures treat the drawing prompt as a neutral vehicle, the author argues that object categories possess inherent properties (e.g., part count, canonical viewpoint, similarity to other objects) that systematically influence scores.

In machine learning, classifiers are known to have skewed label priors (e.g., CLIP inheriting web-scale biases), but this has not been connected to measurement validity in behavioral sciences. The study posits that if stimulus-linked variance exceeds or rivals child-linked variance, the score is not a straightforward measure of the child. Furthermore, if "object difficulty" is actually a property of the scoring instrument's label geometry rather than the referent, standard norming practices (difficulty tables) are non-transportable across different scoring pipelines.

2. Methodology

Data Source:
The study utilizes the KidDraw corpus, comprising 37,770 drawings from 8,084 children (ages 2–10) across 48 object categories. The corpus includes per-session visuomotor control indices (shape-tracing). Two independent machine-vision scoring pipelines were applied:

CLIP Pipeline: Based on CLIP embeddings (zero-shot, web-scale pretraining).
VGG Pipeline: Based on VGG embeddings pretrained on ImageNet (trained on a balanced subset of the drawing corpus).

Statistical Modeling:
The author employed a cross-classified linear mixed model (cross-classified random effects) where children and object categories are crossed rather than nested. This allows for the separate identification of variance components:

$u(i)$ : Child effect ( $\sigma^2_C$ )
$v(k)$ : Object-category effect ( $\sigma^2_K$ )
$\epsilon(i,j,k)$ : Residual error ( $\sigma^2_e$ )

The primary metric of interest is the stimulus-to-child variance ratio ( $R = \sigma^2_K / \sigma^2_C$ ).

Key Metrics and Adjustments:

Developmental Displacement Index (DDI): Rescales stimulus difficulty (in log-odds) into years of chronological development using the estimated age slope ( $\beta_{age}$ ).
Covariate Adjustment: Models were adjusted for visuomotor skill, drawing effort (stroke count, duration, ink), and category familiarity.
Attractor Strength ( $A(k)$ ): A novel metric defined as the mean probability mass a category $k$ draws from drawings of all other categories ( $i \neq k$ ). This isolates the classifier's label geometry (bias) from the child's actual rendering of the target object.
Generalizability Theory (G-Theory): A decision study simulated single-prompt and multi-prompt batteries to calculate dependability coefficients ( $\Phi$ ) and age-ordering error rates.

3. Key Results

Variance Decomposition (Q1 & Q2):

The object category contributes more to score variance than the child. Under the CLIP pipeline, the ratio $R$ is 1.65; under VGG, it is 1.60.
This asymmetry strengthens (ratios > 2.0) after adjusting for visuomotor skill, effort, and familiarity, indicating that controlling for child-level factors does not mitigate the stimulus effect.
DDI: One standard deviation of object difficulty is equivalent to 4.4 to 7.3 years of chronological development. The range between the easiest and hardest categories corresponds to 16.7 to 24.4 years of apparent development, exceeding the entire age span of the sample.

Mechanism of Variance (Q5):

Pipeline Dissociation: While both pipelines agree on the magnitude of stimulus dependence and developmental sensitivity, they disagree entirely on which categories are difficult (correlation of category difficulty estimates $r = 0.205$ , $p = .166$ ).
Attractor Strength: $A(k)$ is a highly stable property of the scoring pipeline (split-half $r = 1.00$ for CLIP). It accounts for 54.3% of the variance in category effects under CLIP.
Unmasking: When category effects are residualized against attractor strength, the cross-pipeline correlation of the remaining "referent-intrinsic" difficulty increases from non-significant ( $r=0.205$ ) to significant ( $r=0.396$ , $p=0.006$ ). This confirms that a large portion of apparent object difficulty is an instrument artifact.

Measurement Cost (Q4):

Reliability: A single-prompt assessment yields an absolute dependability coefficient of only 0.11 to 0.14.
Prompt Requirements: To reach a reliability of 0.80, 26 to 32 prompts are required.
Error Rates: In a single-prompt scenario, an older child (7–10) receives a lower score than a younger child (2–4) approximately 34% of the time, purely due to prompt assignment.

4. Key Contributions

Quantification of Stimulus Dominance: The study demonstrates that in machine-scored drawing assessments, the stimulus (object) explains more variance than the child, challenging the core assumption of these instruments.
Identification of Label Geometry as CIV: The paper identifies "attractor strength" (classifier label geometry) as a major source of construct-irrelevant variance (CIV). It shows that "difficulty" is often a property of where the classifier deposits probability mass, not the object itself.
Non-Transportability of Norms: The findings prove that difficulty norms derived from one scoring pipeline are not transportable to another, rendering standard cross-study comparisons invalid without adjustment.
Methodological Correction: The author proposes and validates "attractor-adjusted" category effects. By removing the pipeline-specific attractor component, a genuine, scorer-independent referent effect is revealed.
G-Theory Application: The study applies generalizability theory to drawing tasks, revealing that single-prompt administration is statistically indefensible for individual-level decisions.

5. Significance and Claims

The paper claims that current machine-scored drawing assessments conflate three superimposed factors: the child, the referent, and the scoring instrument. The author argues that the instrument's label geometry is the second-largest identifiable source of variance, often dominating the child's contribution.

The significance lies in the demonstration that:

Validity Threat: The "window" (the object) is doing more work than the "subject" (the child), and a large part of that work is the "frame" (the classifier's bias).
Practical Necessity: Difficulty norm tables are not fixed constants but are pipeline-specific artifacts. Reporting raw category effects without attractor adjustment leads to non-replicable findings.
Remedy: The field must report attractor-adjusted category effects and treat prompts as random factors (cross-classified designs) rather than fixed items. The author provides code to compute these adjustments using the probability matrices already generated by classifiers.

The study concludes that while drawing tasks contain robust developmental signals, the current practice of reporting a single score as a measure of the child is flawed. Separating the child, referent, and instrument components is not optional but a prerequisite for valid measurement.