Probing Stylistic Appropriation using Large Language Models: An Evaluation Framework for Copyright Infringement under EU Law

Technical Summary: Probing Stylistic Appropriation using Large Language Models

1. Problem Statement

The rapid advancement of Large Language Models (LLMs) has created a critical compliance gap between technical safeguards and EU copyright law. Current technical approaches to copyright infringement focus predominantly on verbatim memorisation (exact or near-exact textual reproduction), utilizing metrics such as n-gram overlap, edit distance, and embedding similarity. However, EU copyright doctrine, particularly under the Copyright in the Digital Single Market (CDSM) Directive and the AI Act, protects works based on originality defined as the "author's own intellectual creation." Infringement determinations extend beyond literal copying to encompass substantial similarity in expressive elements, including narrative structure, character development, stylistic choices, and the arrangement of creative elements.

Existing technical pipelines fail to detect these subtle forms of stylistic and structural appropriation. Consequently, models may pass technical benchmarks designed to prevent literal copying while still generating outputs that appropriate protected expressive choices (e.g., narrative voice, plot architecture) without triggering existing safeguards. This paper addresses the methodological challenge of evaluating LLM outputs at scale for legally relevant expressive similarity when such similarities are qualitative, context-dependent, and non-verbatim.

2. Methodology: The PSALM Framework

The authors introduce PSALM (Probing Stylistic Appropriation by Language Models), an automated evaluation framework that operationalises EU copyright doctrine through an LLM-as-a-judge architecture.

• Architecture: PSALM employs a Directed Acyclic Graph (DAG) structure for each evaluator. The graph consists of:
  • Task Nodes: Analyze specific sub-dimensions of a legal construct (e.g., lexical complexity, focalisation patterns) using structured prompts that encode legal definitions and narratological theories.
  • Judgement Node: Aggregates the outputs of task nodes, applying heuristic weights derived from legal doctrine and narratology to produce a final verdict.
  • Verdict Nodes: Map analyses to a five-level ordinal scale (0 to 10, normalized to 0.0–1.0), representing categories from "clearly different" to "identical or near-identical."
• Evaluators: The framework implements 10 evaluators divided into two categories:
  1. Infringement-Oriented (6 evaluators): Measure similarity across protectable expressive features:
     • Writing Style: Lexical complexity, sentence structure, rhythm, rhetorical patterns, discourse organisation, tone.
     • Narrative Voice: Point of view, narrative distance, narrator presence, temporal perspective, focalisation, reader engagement.
     • Character: Identity/traits, arc/development, relationships, background/motivation, expression/behaviour, function/role.
     • Plot Structure: Event sequence/causality, story architecture, conflict construction, turning points, temporal structure, convergence.
     • Scene Sequence: Identity/content, internal structure, sequence architecture, transitions, pacing, functions.
     • World Building: Geographic/spatial design, rules/systems, cultural/social architecture, historical/temporal design, material/sensory detail, function/logic.
  2. Defence-Oriented (4 evaluators): Assess whether similarities qualify for statutory exceptions or fall within unprotectable elements:
     • Parody/Satire: Based on the Deckmyn test (evocation, transformation, humour, critical character, fair balance).
     • Pastiche: Based on Pelham II and AG Emiliou's opinion (style evocation, artistic skill, homage, transformation, proportionality).
     • Quotation/Citation: Based on Art. 5(3)(d) InfoSoc (identification, legitimate purpose, fair practice, attribution, prior disclosure).
     • Scènes à Faire: Assessing stock elements dictated by genre or technical constraints.

3. Experimental Setup

The study evaluates the PSALM framework using Llama 3.2 1B Instruct models fine-tuned on a corpus of translated historical Dutch literary works (from the Digital Library of Dutch Literature).

• Data Construction: Historical Dutch texts were converted into a Question-Answer (QA) dataset to emphasise stylistic learning rather than prompt copying. Texts were translated into modern Dutch and English.
• Training Regimes:
  • Baseline: Unmodified instruction-tuned models.
  • Fine-tuned: Models trained on the full QA dataset (forget + retain sets).
  • Unlearned: Fine-tuned models subjected to Negative Preference Optimisation (NPO) to "unlearn" specific authors or books (forget sets) while retaining others (retain sets).
• Evaluation: The framework compares model outputs against source texts using PSALM evaluators and standard lexical baselines (Exact Match, BLEU-4, ROUGE-L). Statistical analysis employs non-parametric tests (Kruskal-Wallis, Mann-Whitney, Wilcoxon) and equivalence testing (TOST).

4. Key Results

The experiments address three research questions regarding baseline similarity, the impact of fine-tuning, and the efficacy of unlearning:

• RQ1: Baseline Stylistic Similarity: Instruction-tuned base models exhibit non-trivial baseline stylistic similarity to protected works prior to corpus exposure, particularly in English models which show higher pre-training overlap with late 19th-century prose styles. However, these scores generally fall into "somewhat similar" or "moderately similar" bands, not "near-identical."
• RQ2: Impact of Fine-Tuning: Supervised fine-tuning induces systematic stylistic appropriation across all infringement-relevant dimensions.
  • Fine-tuned models achieve near-ceiling scores (0.95–1.0) on computational, stylistic, and content metrics (Writing Style, Narrative Voice, Character, Plot, World Building).
  • This appropriation extends far beyond verbatim memorisation; the models reproduce abstract narrative patterns and authorial voices.
  • Crucially, fine-tuning does not increase scores on defence-oriented metrics (Parody, Pastiche, Quotation), indicating that the models do not spontaneously adopt transformative or exception-qualifying behaviours.
• RQ3: Effectiveness of Unlearning (NPO): Negative Preference Optimisation substantially reduces similarity but fails to fully restore baseline behaviour.
  • NPO effectively eliminates verbatim memorisation (Exact Match drops to zero).
  • Stylistic and content similarity scores decrease significantly (e.g., from ~0.99 to ~0.30 on forget sets) but remain detectable and often fall into the "somewhat different" to "moderately similar" range.
  • Residual Similarity: Equivalence testing (TOST) confirms that unlearned models are not statistically equivalent to baseline models. Residual stylistic patterns persist, particularly on retain sets where scores remain in the "very similar" region.
  • Language effects are limited, though Dutch models showed slightly higher residual world-building similarity than English models.

5. Significance and Claims

The paper claims to bridge the gap between qualitative legal standards and quantitative technical measurement, exposing fundamental tensions between generative AI and EU intellectual property law.

• Methodological Contribution: PSALM provides the first infrastructure for auditable, legally informed compliance evaluation that moves beyond surface-form overlap to assess structural and stylistic appropriation. It demonstrates that LLMs can approximate the context-dependent reasoning of human legal evaluators when grounded in specific doctrinal criteria.
• Legal Implications: The findings indicate that safeguards targeting literal copying alone are insufficient to mitigate broader copyright risks. Fine-tuning can create models that act as precision tools for reproducing specific works' expressive elements, while current unlearning techniques (NPO) offer only partial mitigation, leaving detectable residual stylistic patterns that may still constitute substantial similarity under EU law.
• Modesty and Limitations: The authors explicitly state that PSALM scores inform rather than replace human legal judgement. The validation relies on researcher-designed test cases to establish internal consistency and measurement stability, not legal validity against real-world court adjudications. The relationship between automated similarity scores and actual infringement determinations requires further validation by legal experts. Furthermore, the study notes that the QA format may represent an upper bound on stylistic appropriation compared to more varied instruction-tuning pipelines.

In summary, the work demonstrates that while technical unlearning can suppress verbatim extraction, it does not fully erase the "imprint" of stylistic and structural learning, leaving a significant compliance gap for model developers and rightsholders in the era of generative AI.