Can You Make It Sound Like You? Post-Editing LLM-Generated Text for Personal Style

1. Problem Statement

While Large Language Models (LLMs) are increasingly used for professional writing, their adoption in personal writing tasks (e.g., wedding vows, apology letters, eulogies) remains controversial. Users often fear that AI assistance compromises authenticity and personal style.

Prior research suggests users are willing to use AI for planning but resist it during the "translation" (drafting) and "reviewing" stages, as these directly shape stylistic expression. This paper investigates a specific collaborative workflow: Post-Editing. The core research question is: Can users effectively post-edit LLM-generated drafts to reintroduce their unique personal style, and does this process result in text that is perceptually and technically indistinguishable from their unassisted writing?

2. Methodology

Study Design

The authors conducted a pre-registered, IRB-approved online study with $N=81$ participants (recruited from Prolific, filtered for English fluency).

• Tasks: Participants selected 6 writing tasks from a pool of 8 scenarios where personal style is critical (e.g., wedding vows, apology letters, condolence letters).
• Conditions: The study utilized a within-subjects design with two blocks:
  1. Control Block: Participants planned details and wrote the text entirely from scratch (unassisted).
  2. Treatment Block: Participants planned the same details, received an LLM-generated draft (based on their plan), and then post-edited the draft to reflect their personal style.
• Metrics:
  • Stylistic Similarity: Measured using LUAR embeddings (Rivera-Soto et al., 2021), specifically the LUAR-MUD model, which captures authorship style via cosine similarity.
  • AI Detection: Measured using PANGRAM scores to estimate the likelihood of text being AI-generated.
  • Linguistic Acceptability: Evaluated using a CoLA-based model to assess grammatical correctness.
  • Self-Reports: Participants rated perceived stylistic authenticity, usability, and task difficulty.

Hypotheses

The study tested three main hypotheses:

• H1 (Authenticity): Post-editing will make LLM text sound more like the user's unassisted writing and less like raw LLM output.
• H2 (Homogeneity): Post-edited text will form a distinct "third" style—more homogeneous than human writing but less than raw LLM output.
• H3 (Perception): User perception of stylistic similarity will align with model-measured similarity.

3. Key Contributions & Results

A. Effectiveness of Post-Editing (H1)

• Statistical Shift: Post-editing significantly increased stylistic similarity to the participant's own unassisted control text ($p < .001$, $g=0.55$) and significantly decreased similarity to raw LLM text ($p < .001$, $g=-0.41$).
• Residual AI Traces: Despite these improvements, post-edited text remained significantly more similar to LLM-generated text than to the participant's own unassisted writing ($p < .001$, $g=-1.43$). Users could not fully eliminate the "LLM fingerprint."
• Specificity: The edits made were specific to the individual. Post-edited text was more similar to the editor's own style than to other participants' styles, confirming that the changes were not just generic "humanization."

B. Stylistic Homogeneity (H2)

• The "Third Style": Post-edited text exhibited a unique stylistic profile. It was more homogeneous (less diverse) than fully human writing but less homogeneous than raw LLM output.
• Shared AI Traits: While individual edits were unique, the un-edited portions of the text retained shared LLM stylistic features across all participants. This resulted in post-edited texts from different users being more similar to each other than to their own unassisted control texts.

C. The Perception Gap (H3)

• Disconnect: There is a significant disconnect between model-measured similarity and user-perceived authenticity.
  • Model View: Post-edited text is still clearly "AI-like" compared to human control text.
  • User View: Participants rated their post-edited text as equally representative of their personal style as their unassisted control text ($p = .906$).
• Correlation: While there was a positive correlation between perceived and measured similarity ($r=0.24$), it was weakly calibrated. Users did not perceive the residual stylistic markers that the embedding models detected.

D. Editing Behavior Analysis

• Edit Types: Participants employed diverse strategies, including:
  • Lexical changes: Removing "AI-associated" words (e.g., "delve," though rare in this dataset) or adding contractions (5x more frequent in post-edited text).
  • Punctuation: Removing em-dashes (23% removal rate) and adding non-standard punctuation or typos to mimic personal habits.
  • Semantic edits: Correcting factual details or adding culturally specific language (e.g., religious phrasing).
• Effort vs. Outcome: The study found that originality of the initial draft was the primary barrier to adoption. Participants who preferred writing alone cited a lack of originality in AI drafts, whereas those who preferred post-editing valued efficiency and reliability.

4. Significance and Implications

1. The "Authenticity Illusion": The paper highlights a critical gap where users feel their AI-assisted writing is authentic, even when objective metrics detect significant AI stylistic traces. This suggests users may rely on simplified heuristics (e.g., "did I change the words?") rather than deep stylistic analysis.
2. Workflow Viability: Despite the inability to fully erase AI stylistic markers, the workflow is viable. Users report high satisfaction and a willingness to use post-editing for future personal tasks, provided the initial draft is not too clichéd.
3. Limitations of Current Metrics: Embedding-based style metrics may capture statistical salience that does not align with what lay users consider "style." This challenges the utility of current AI detectors in personal writing contexts where the goal is user satisfaction rather than forensic detection.
4. Future Directions: The authors suggest that improving the creativity and originality of initial LLM drafts is more impactful than trying to teach users to post-edit more effectively. Additionally, future work must consider the audience's perspective: will readers (who know the author's style) detect the residual AI markers that the authors missed?

Conclusion

The study concludes that while post-editing allows users to exert meaningful control and achieve a sense of stylistic authenticity, it does not fully bridge the gap between LLM and human writing styles. The resulting text is a hybrid: statistically closer to the model than the human, but perceived by the user as authentically their own. This disconnect underscores the need for better alignment between AI generation capabilities and human stylistic expectations, rather than relying solely on post-hoc editing.