Technical Summary: Simplifying the Modeling of Arbitrary Conditionals in Natural Language

Problem Statement

Standard Causal Transformers, the architecture underpinning modern Large Language Models (LLMs), model sequences via an autoregressive factorization of the joint distribution ($p(x) = \prod_t p(x_t | x_{<t})$). This left-to-right structure enables efficient training and decoding but fundamentally restricts the model's ability to tractably evaluate or sample from arbitrary conditionals. Specifically, these models cannot compute the likelihood of a sequence segment conditioned on a mix of past and future tokens (e.g., $p(x_{middle} | x_{past}, x_{future})$) in a single forward pass.

Existing approaches to this problem include:

• Any-order Autoregressive Models (e.g., XLNet, $\sigma$-GPT): These modify the architecture to support arbitrary token orderings. However, they often require complex architectural changes (e.g., double positional encodings) and waste model capacity by learning to predict tokens under all possible orderings, including sub-optimal ones for natural language.
• Masked Language Models (MLMs): While capable of bidirectional attention, they maximize the likelihood of independent marginals rather than joint probabilities, making them non-generative and computationally expensive for sequential sampling (requiring iterative unmasking).
• Discrete Diffusion Models: These can sample from arbitrary conditionals but generally struggle to tractably evaluate likelihoods and often underperform autoregressive models in perplexity due to a lack of left-to-right inductive bias.

Crucially, most of these alternatives require training models from scratch, making it infeasible to adapt existing, pre-trained LLMs to handle arbitrary conditioning without significant architectural surgery.

Methodology: Arbitrary Conditionals GPT (AC-GPT)

The authors propose AC-GPT, a method that augments standard causal Transformers to handle arbitrary conditionals while preserving their original architecture and training objectives.

Core Mechanism

AC-GPT introduces a simple modification to the input sequence representation:

1. Augmentation with Conditioned Copies: For a given conditional query where a sequence $x$ is partitioned into a conditioning set $x_c$ and an evaluation set $x_e$, the model creates copies of the conditioning tokens, denoted as $\bar{x}_c$.
2. Sequence Construction: These copies are placed at the "start" of the sequence (conceptually before the original tokens) such that all other tokens in the sequence can attend to them.
3. Attention Masking:
   • Within $\bar{x}_c$: Unrestricted bidirectional attention is used, as these tokens are known and do not need to respect causal ordering relative to each other.
   • Rest of the Sequence: Standard causal attention is maintained. Tokens in the evaluation set $x_e$ can attend to $\bar{x}_c$ and previous tokens in $x_e$, but cannot attend to future tokens in $x_e$.
4. Positional Integrity: The copies $\bar{x}_c$ retain the positional encodings of their original locations in the sequence. This ensures the model understands the relative position of the conditioned tokens within the original context.
5. Preventing Information Leakage: The copying mechanism is critical to prevent information leakage across Transformer layers. Without it, standard causal attention would allow information from later tokens in the evaluation set to indirectly influence earlier tokens through the conditioning tokens in subsequent layers, violating the chain rule decomposition.

Training Strategy

• Objective: The model is trained using the standard next-token prediction objective (teacher forcing). The loss is computed only on the tokens in the evaluation set $x_e$. Predictions for the conditioning tokens $\bar{x}_c$ are ignored.
• Conditioning Distribution: To enable inference on truly arbitrary queries, the model is trained on a distribution of conditioning sets. The authors sample conditioning sets $x_c$ by interpolating between fully arbitrary queries and structured queries (e.g., contiguous blocks). This involves sampling the size of the conditioning set, the number of contiguous blocks, and their locations.

Key Contributions

1. Architectural Simplicity: AC-GPT requires no fundamental changes to the causal Transformer architecture (e.g., no double positional encodings or custom attention masks beyond the standard causal mask). It simply augments the input sequence.
2. Single-Pass Evaluation: Unlike MLMs or diffusion models, AC-GPT can evaluate the joint likelihood of an arbitrary conditional in a single forward pass.
3. Preservation of Left-to-Right Inductive Bias: By maintaining the standard next-token prediction objective and causal ordering for the evaluation set, AC-GPT retains the strong performance characteristics of standard LLMs on natural language tasks.
4. Fine-Tuning Compatibility: Because the architecture remains a standard causal Transformer, AC-GPT allows for the direct fine-tuning of pre-trained LLMs (at the billion-parameter scale) to gain arbitrary conditioning capabilities, a feat not possible with $\sigma$-GPT or other any-order models that require training from scratch.

Experimental Results

The authors evaluated AC-GPT on the Wikitext-103 and FineWeb datasets, comparing it against standard GPT, $\sigma$-GPT (random and temporal variants), MLMs (BERT), and Diffusion models.

• Performance on Conditional Queries: AC-GPT significantly outperformed baselines on conditional likelihood evaluation and sampling tasks (e.g., infilling, training distribution queries).
  • On Wikitext-103, AC-GPT achieved a perplexity of 17.1 on infilling tasks, compared to 45.4 for BERT and catastrophic failure for $\sigma$-GPT-random.
  • It demonstrated a 15–16% reduction in perplexity on training-distribution queries when leveraging future context compared to models restricted to past context.
• Preservation of Standard Performance: AC-GPT maintained competitive performance on unconditional (left-to-right) generation, with perplexity scores nearly identical to the standard GPT baseline (e.g., 18.4 vs. 17.9).
• Scaling to Pre-trained Models: In fine-tuning experiments with Qwen3 models (8B, 14B, 32B) and LLaMA-3.1 (8B), AC-GPT LoRA adapters achieved substantial gains (13–15% perplexity reduction) on conditional tasks without degrading unconditional performance. The gains were consistent across model scales and adapter capacities (LoRA vs. full fine-tuning).
• Sampling Quality: MAUVE scores indicated that AC-GPT and $\sigma$-GPT-temporal produced high-quality generations comparable to the vanilla GPT baseline, whereas $\sigma$-GPT-random and Diffusion models often collapsed into incoherent text or "word salad" in conditional settings.

Significance and Claims

The paper claims that AC-GPT offers a paradigm-level solution to the problem of arbitrary conditioning by decoupling the modeling of arbitrary conditionals from the modeling of arbitrary token orderings.

• Efficiency vs. Compatibility: The authors acknowledge a trade-off: AC-GPT sacrifices some training efficiency (by only backpropagating loss through the evaluation set) to gain architectural compatibility with pre-trained models. They argue this is a favorable trade-off given the prevalence of pre-trained LLMs.
• Practical Impact: The primary significance lies in the ability to augment existing, powerful LLMs with the capability to reason about future context and perform tasks like text infilling and editing, without the prohibitive cost of training new architectures from scratch.
• Modesty: The authors note that their current work focuses on conditioning on known tokens. They explicitly state that marginalizing over unknown sets of tokens (a different type of query) is not yet explored in this work, though they suggest it could be achieved by introducing a MASK token. They also acknowledge that their experiments were restricted to two datasets and that future work should explore larger corpora and specific downstream applications.

In summary, AC-GPT demonstrates that the ability to handle arbitrary conditionals does not require abandoning the successful left-to-right autoregressive framework, but rather a simple, position-aware augmentation of the input context.