Know Your Author: Does the AI Penalty Hold in Short Fiction?

Technical Summary: Does the AI Penalty Hold in Short Fiction?

Problem Statement
Public discourse and emerging literature suggest an "AI penalty," positing that labeling content as AI-generated leads to more negative evaluations compared to human-created work, particularly in aesthetic domains. However, existing evidence often conflates authorship beliefs with intrinsic differences in the artifacts themselves, as studies frequently compare distinct human and AI outputs. This creates ambiguity regarding whether observed evaluation gaps stem from bias against the source or actual quality disparities. Furthermore, the role of the "effort heuristic"—the tendency to infer quality from perceived labor—and the moderating influence of prior attitudes toward AI remain under-explored in controlled settings where content is held constant.

Methodology
The authors conducted a preregistered, between-subjects experiment ($N = 254$ per protocol) using a pure attribution design to isolate the causal effect of authorship labels.

• Stimuli: Two ~200-word short fiction vignettes ("Firefly Summer" and "The Endless Summer") were generated by a single AI system (Anthropic's Claude 3.7 Sonnet) from an identical prompt.
• Design: A 3 (Attribution Label: Human-written, AI-written, Unlabeled) $\times$ 2 (Story) design. Participants were randomly assigned to read one vignette with one of three authorship lines:
  • Human: "Author: J. Taylor, an American short fiction writer."
  • AI: "Author: Claude, an AI system specialized in creative writing."
  • Unlabeled: No author information.
• Measures:
  • Primary Outcomes: Ratings of creativity, enjoyment, recommendation likelihood, and originality on 1–7 Likert scales.
  • Perceived Effort: Open-ended estimates of the time taken to create the story (converted to minutes).
  • Moderators: Single-item measures of AI familiarity and general positivity toward AI.
• Analysis: Confirmatory analyses utilized one-way and two-way ANOVAs to test for main effects and interactions. Effort heuristic relationships were tested via Pearson correlations and OLS regression using log-transformed time estimates ($\ln[\text{minutes} + 1]$) to address skewness. Moderation effects were analyzed via regression interactions.

Key Results

1. Absence of Direct AI Penalty: Contrary to Hypothesis 1 (H1) and H2, authorship labels did not produce reliable main effects on creativity, enjoyment, recommendation, or originality. Effect sizes were uniformly small and statistically non-significant ($p > .22$). Notably, the Unlabeled condition received the lowest average evaluation scores, contrary to the prediction that it would fall between Human and AI conditions.
2. Strong Effect on Perceived Effort: While labels did not alter average quality ratings, they drastically influenced inferred effort. Participants estimated Human-labeled stories took significantly longer to create (geometric mean $\approx$ 148 minutes) compared to AI-labeled stories ($\approx$ 6 minutes). Unlabeled stories were estimated at $\approx$ 75 minutes.
3. Effort Heuristic Validation: Supporting H3, higher perceived creation time was positively associated with higher enjoyment ratings. This relationship held across all conditions and specifically within the AI-labeled group ($r = .37, p = .001$). Mediation analyses confirmed that AI labeling predicted lower perceived time, which in turn predicted lower enjoyment, though direct effects remained near zero.
4. Moderation by AI Attitudes: Prior attitudes toward AI significantly moderated recommendation judgments. Participants with positive attitudes toward AI were more likely to recommend AI-labeled stories, whereas those with negative attitudes showed a preference for Human-labeled stories. This interaction suggests that average null effects may result from opposing subgroup trends canceling each other out.
5. Manipulation Check: Authorship recall was imperfect; 14.3% of participants failed to correctly identify the assigned label, with failures concentrated in the Human-labeled condition.

Significance and Claims
The paper claims that in the context of short fiction, explicit AI authorship labels do not systematically degrade average evaluations of the text itself when the content is held constant. The "AI penalty" is not a universal, direct penalty on quality perception in this setting.

Instead, the study identifies two mechanisms through which labels operate:

1. Effort Inference: Labels shape perceptions of the labor involved in creation. The "AI penalty" may be mediated by the assumption that AI requires minimal effort, which subsequently lowers enjoyment, rather than a direct devaluation of the aesthetic output.
2. Attitudinal Moderation: The impact of labeling is contingent on the reader's pre-existing beliefs about AI.

The authors conclude that if an AI penalty exists in this domain, it is likely small and context-dependent. They note limitations, including the use of short, uplifting vignettes and a specific US-based sample, suggesting that larger replications with more precise measurements are required to determine if the true effect is a small penalty, effectively zero, or potentially a slight benefit of explicit attribution. The study emphasizes that disclosure influences downstream judgments less through a direct penalty and more through the salience of the label and the effort inferences it triggers.