LLM Output Detectability and Task Performance Can be Jointly Optimized

Technical Summary: PUPPET – Jointly Optimizing LLM Output Detectability and Task Performance

Problem Statement
The deployment of Large Language Models (LLMs) raises urgent concerns regarding misuse, including academic fraud, spam, and automated disinformation. While detecting machine-generated text is essential for transparency, current approaches face a significant trade-off. Watermarking methods, which bias token distributions to enable statistical detection, are the de facto standard. However, existing literature reports that optimizing solely for detectability often degrades performance on downstream tasks. Conversely, optimizing solely for task performance yields text that is difficult to distinguish from human writing. There is a lack of methods that can simultaneously achieve high detectability and strong task performance without compromising one for the other.

Methodology: PUPPET
The authors propose PUPPET (Preference Optimization for Unified Performance and Detectability), a framework that fine-tunes an LLM using Direct Preference Optimization (DPO) to jointly optimize for both detectability and task performance.

1. Preference Data Construction:

   • For a given input query $q$, the base model $\pi$ generates $k$ candidate responses $\{t_1, \dots, t_k\}$.
   • Each response is scored by two distinct functions:
     • Detector Score ($S_{Det}$): The probability of the text being machine-generated, computed by an external detector (e.g., OpenAI Detector).
     • Evaluator Score ($S_{Eval}$): A task-specific metric (e.g., ROUGE-L for summarization, LLM-as-a-Judge for essay writing).
   • These scores are normalized via z-score and combined into a final score $S_j = \alpha \cdot z(S_{Det}) + (1-\alpha) \cdot z(S_{Eval})$, where $\alpha$ controls the trade-off.
   • The response with the highest combined score is selected as the "chosen" response, and the lowest as the "rejected" response.

2. Training:

   • The model is fine-tuned using DPO on the constructed preference dataset. DPO is chosen for its training stability compared to policy-gradient methods like PPO.
   • The process requires only a few thousand samples and can be completed in 1–2 GPU hours using LoRA for memory efficiency.

Key Contributions and Results
The paper evaluates PUPPET on Llama-3-8B-Instruct and Qwen3 variants across three tasks: long-form QA (ELI5), summarization (Multi-News), and essay writing (IELTS). The results are compared against five state-of-the-art watermarking baselines (KGW, SynthID, Unigram, EXPGumbel, K-SemStamp) and a vanilla (unmodified) baseline.

• Simultaneous Optimization: PUPPET successfully improves both objectives relative to the vanilla baseline. For instance, on long-form QA, it improved detectability (AUROC) by +3.8 points and task performance (ROUGE-L) by +2.6 points on average over watermarking methods.
• Superiority to Watermarking: While watermarking methods often suffer task performance degradation, PUPPET achieves task performance levels that watermarked models cannot reach, while matching or surpassing them in detectability.
• Generalizability: The improvements transfer to out-of-domain tasks. A model trained on one task (e.g., ELI5) significantly improves detection on unseen tasks (e.g., Multi-News) without degrading performance.
• Robustness:
  • Model Family/Size: Gains are consistent across different model families (Llama-3, Qwen3) and sizes (4B to 14B).
  • Detector Independence: The method works effectively with different detectors (OpenAI, FakeSpotAI, MAGE).
  • Paraphrasing Attacks: Unlike watermarking methods, which suffer severe AUROC drops (up to 49.2 points) under paraphrasing attacks (e.g., Dipper), PUPPET shows near-zero degradation (average drop of 1.2 points). The authors attribute this to the model learning deeper, structural features rather than surface-level token patterns.
• Efficiency: The method is data-efficient, converging within 1,000–2,000 samples, and computationally lightweight.
• Model Attribution: Exploratory analysis suggests PUPPET enables model attribution, allowing the distinction of PUPPET-trained models from other models based on the detector's likelihood scores.

Significance and Claims
The paper claims that PUPPET addresses a critical limitation in current LLM accountability mechanisms: the trade-off between detectability and utility. By demonstrating that these objectives can be jointly optimized, the authors argue that it is possible to deploy LLMs that are both highly useful and transparently identifiable as machine-generated.

The authors position this work as a step toward more transparent LLM deployment. They note that while watermarking is currently the standard, it is not the only path; direct optimization via preference learning offers a viable alternative that avoids the performance penalties associated with distributional biasing. The work concludes by identifying future directions, including deeper mechanistic explanations of the learned features and systematic hyperparameter tuning, while maintaining a cautious stance on the research stage of these detection technologies.