Teaching LLMs Human-Like Editing of Inappropriate Argumentation via Reinforcement Learning

1. Problem Definition

Large Language Models (LLMs) are increasingly used to edit text, particularly to improve the appropriateness of arguments (e.g., removing toxic language, emotional appeals, or unclear reasoning). However, a critical mismatch exists between human and LLM editing strategies:

• LLM Behavior: Often generates "scattered" edits that change the meaning of the original text significantly or produce interdependent changes that cannot be applied selectively.
• Human Behavior: Produces self-contained, meaning-preserving edits. Humans encapsulate dependent changes into single, contiguous operations that maintain semantic similarity, fluency, and follow typical edit patterns (keep/delete/add/substitute).

The paper addresses the challenge of teaching LLMs to generate human-like edit suggestions for inappropriate arguments that are:

1. Semantically Similar: Preserving the original meaning.
2. Fluent: Not introducing grammatical errors.
3. Conformant: Following typical human edit patterns (e.g., avoiding massive rewrites for minor fixes).
4. Appropriate: Successfully mitigating the identified inappropriateness.

2. Methodology

The authors propose a Reinforcement Learning (RL) framework using Group Relative Policy Optimization (GRPO) to train an LLM policy to generate these specific types of edits.

A. Policy and Objective

• Policy ($\pi$): An LLM (specifically Llama-3.1-8B-Instruct) that takes an argument $a$ and outputs a set of edit suggestions $E = \{e_1, ..., e_k\}$. Each edit $e_i$ consists of a text span $s_i$ and its replacement $t_i$.
• Training Algorithm: GRPO is used for its memory efficiency compared to standard PPO. It optimizes the policy parameters $\theta$ to maximize the expected reward $R(a, E)$.

B. Reward Function Design

The core innovation is a multi-component reward function that balances edit-level quality with argument-level appropriateness:
$$R(a, E) = \alpha \cdot r_{arg}(a, E) + (1 - \alpha) \cdot r_{edit}(E, a)$$

1. Edit-Level Reward ($r_{edit}$): Measures "Human-Likeness." An edit is considered human-like only if it passes three binary classifiers:

   • Semantic Similarity ($c_{sim}$): Uses EmbeddingGemma-300M to ensure the edited sentence is semantically close to the original (threshold set at the 99th percentile of human edits).
   • Fluency ($c_{flu}$): A ModernBERT classifier trained to detect if an edit degrades grammatical correctness. It is optimized for high precision to reject edits that break coherence.
   • Pattern Conformity ($c_{con}$): A language model trained on edit operation sequences (Keep, Delete, Add, Substitute) from the IteraTeR dataset. It evaluates the perplexity of the edit sequence; low perplexity indicates the edit follows typical human patterns.
   • Calculation: $r_{edit}$ is the proportion of edits in the set that pass all three criteria.

2. Argument-Level Reward ($r_{arg}$): Measures the improvement in appropriateness.

   • An appropriateness classifier (trained on 14 dimensions of inappropriateness) evaluates the argument after applying only the human-like edits.
   • This ensures the model is incentivized to actually fix the argument, not just make "safe" edits.

C. Data

• Human Revisions: The IteraTeR dataset (Wikipedia, ArXiv, Wikinews) is used to train the similarity, fluency, and conformity classifiers.
• Appropriateness Data: The Ziegenbein et al. (2023) corpus, extended with soft-labeled arguments from the Internet Argument Corpus and GAQCorpus, provides the training data for the GRPO policy.

3. Key Contributions

1. First RL Approach for Human-Like Edits: Unlike previous methods that generate full rewrites or rely on supervised fine-tuning with parallel data, this is the first RL approach explicitly optimizing for edit-level quality (similarity, fluency, pattern) without requiring parallel human-LLM edit pairs.
2. Novel Reward Modeling: Introduces a Pattern Conformity metric that evaluates the sequence of edit operations (diffs) against human behavior, alongside standard semantic and fluency metrics.
3. Self-Contained Edit Generation: The system produces granular, independent edit suggestions that users can accept or reject individually, unlike "black box" full rewrites.

4. Results

The approach was evaluated against baselines (including PPO-based rewriting methods and direct LLM prompting) on a test set of 225 inappropriate arguments.

• Edit-Level Performance:
  • The full model (GRPOfull) generated 1,221 human-like edits (66.9% of total edits), significantly outperforming the best baseline (PPO-based) which generated only 239 (34.7%).
  • It achieved the highest scores in Semantic Similarity (0.915), Fluency (0.757), and Pattern Conformity (0.939).
• Argument-Level Performance:
  • While baselines often sacrificed human-likeness to improve appropriateness, GRPOfull maintained a strong balance, achieving a 36.4% rate of converting inappropriate arguments to appropriate ones using only human-like edits.
  • Iterative Editing: When applied iteratively (re-editing the output), the model approached the appropriateness levels of full-rewrite baselines while retaining human-like edit characteristics.
• Human Evaluation:
  • In pairwise comparisons, human annotators ranked the GRPOfull (11th iteration) arguments as the best in 82% of cases, significantly outperforming PPO baselines.
  • The model received the highest Likert scores for similarity, fluency, and overall human-likeness.

5. Significance and Implications

• Bridging the Gap: The paper successfully demonstrates that LLMs can be aligned to mimic human editing strategies (minimal, self-contained changes) rather than just optimizing for document-level metrics.
• Actionable Feedback: By generating discrete, selectable edits, the system supports human-in-the-loop workflows, allowing users to refine arguments without losing their original voice or intent.
• Ethical Considerations: The authors emphasize that "appropriateness" is subjective and culturally dependent. They recommend human oversight to prevent "tone policing" or the suppression of marginalized discourse styles, positioning the tool as an assistant for refinement rather than an automated censor.
• Future Work: The paper identifies limitations in handling discourse-level structural reorganization (e.g., reordering paragraphs) and the transferability of these edit patterns to non-English languages.

In conclusion, this work establishes a new standard for LLM-assisted text optimization, proving that explicit reward modeling for edit-level quality yields significantly more usable and human-aligned results than traditional rewriting approaches.