💬 NLP

Cross-Domain, Multi-Task Data-to-Text Generation without In-Domain Training Data

This paper proposes a data-driven knowledge distillation (DDKD) approach with structure-preserving augmentation to achieve effective cross-domain, multi-task data-to-text generation without in-domain training data, demonstrating that small distilled models consistently outperform zero-shot inference and out-of-domain fine-tuning across five benchmarks.

Original authors: Yifei Song, Kun Efimov-Zhang, Claire Gardent

Published 2026-08-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yifei Song, Kun Efimov-Zhang, Claire Gardent

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where computers can read a spreadsheet of weather data and write a readable forecast, or look at a chart of vaccination rates and describe the trends in a news article. This ability, known as data-to-text generation, is already transforming how we interact with information, turning cold numbers into human stories. For years, researchers have taught computers to do this by showing them thousands of examples where a specific table or graph was paired with a perfect description. The computer learns the pattern and eventually writes its own. However, this approach hits a wall when the computer encounters a new type of data it has never seen before, like a specialized medical record or a unique sports statistic, for which no human-written examples exist. Without these examples, the computer often stumbles, producing text that is either nonsensical or factually wrong.

A team of researchers set out to solve this problem by teaching computers to learn from scratch, without any pre-written examples for the new tasks. They focused on a challenging scenario where the computer must generate text for five completely different domains—ranging from ice hockey game summaries to weather reports and product specifications—using only the raw data itself. The goal was to see if a small, efficient computer model could learn to write accurately about these diverse topics without needing a massive library of training texts. Instead of relying on the computer's existing knowledge alone, or trying to force it to memorize a different type of data, the team developed a method where a large, powerful computer acts as a teacher. This teacher generates sample descriptions for the new data, which a smaller student model then learns from. Crucially, the researchers found that simply giving the student more raw data wasn't the best path; instead, they taught the student by breaking the complex data into smaller, simpler pieces and slightly varying them, creating a rich and diverse set of practice examples.

The researchers tested this approach using five distinct real-world datasets, including time-series weather forecasts, knowledge graphs about historical entities, and detailed specifications for mobile phones. They compared three different ways of teaching the computer: letting it guess based on its pre-existing training, teaching it with examples from a completely different field, and using their new method of data-driven knowledge distillation. The results were striking. The small computer models, which have about 1.7 billion parameters, consistently produced fewer errors than both the zero-shot guessing and the models trained on unrelated data. In fact, these small, distilled models outperformed much larger models with 32 billion parameters on four of the five domains. The key to this success was not just the size of the teacher, but how the training data was prepared. By taking the raw data and systematically creating variations—such as removing some details to make the task easier or slightly changing values to prevent the model from memorizing specific numbers—the researchers created a more robust learning environment. This strategy allowed the small models to generalize better, handling the complexity of long, dense data inputs without getting confused.

The study also addressed a common concern in artificial intelligence: that a model might produce fewer errors simply by saying less, omitting important details to avoid mistakes. The researchers checked this by measuring how much of the original information the generated text actually covered. They found that the improved accuracy did not come at the cost of missing information; the models remained faithful to the data while covering the essential points. This suggests that the method successfully teaches the computer to understand the structure of the data and translate it accurately, rather than just playing it safe. The team also constructed a larger version of their test data to see if simply collecting more real-world examples would yield better results. They discovered that while adding more data helped, the strategy of creating structured variations from a smaller set was more effective and cost-efficient. This finding implies that for many real-world applications where data is abundant but human-written descriptions are scarce, the most efficient path forward is not just to gather more raw numbers, but to teach computers how to learn from the patterns within those numbers using smart, synthetic training techniques.

Ultimately, this work demonstrates that small, specialized computer models can be trained to handle complex, unfamiliar data tasks without the need for massive, task-specific datasets. By using a large model to generate synthetic examples and carefully varying those examples to cover different structural possibilities, the researchers enabled a compact model to rival the performance of much larger systems. This approach offers a practical solution for deploying data-to-text generation in diverse fields where collecting human-written training data is impossible or too expensive. The findings suggest that the future of this technology lies not in making models infinitely larger, but in teaching them to learn more efficiently from the data they already possess, turning raw information into reliable, human-readable narratives across any domain.

Technical Summary: Cross-Domain, Multi-Task Data-to-Text Generation without In-Domain Training Data

Problem Definition

Structured data exists in diverse forms (tables, knowledge graphs, time series, charts) and domains (weather, sports, health, technical specifications). Converting this data into natural language (Data-to-Text or D2T) typically requires task-specific training data. However, in many real-world scenarios, in-domain training text and test references are unavailable.

Existing approaches face significant challenges in this setting:

Schema and Format Heterogeneity: Input structures vary substantially in fields, nesting depth, and attribute semantics (e.g., JSON vs. CSV vs. Markdown), making unified representations difficult.
Data Scarcity: The absence of target-domain references precludes conventional supervised fine-tuning (SFT) and reference-based evaluation.
Context Limitations: Structured inputs in domains like weather forecasting or chart captioning are often long and information-dense, rendering few-shot prompting impractical due to context window limits.
Limited Zero-Shot Generalization: Prior work indicates that zero-shot Large Language Models (LLMs) often fail to generate factually accurate text across diverse domains, frequently producing hallucinations or semantic errors.

The paper investigates how to generate fluent, faithful English text from heterogeneous data structures across five distinct domains without any target-domain reference text.

Methodology

The authors propose a framework centered on Data-Driven Knowledge Distillation (DDKD) combined with Structure-Preserving Augmentation.

1. Data-Driven Knowledge Distillation (DDKD)

Instead of directly fine-tuning a small model on scarce target data or relying solely on zero-shot prompting, the authors transfer capabilities from a large "teacher" model to a compact "student" model (1.7B parameters) using synthetic data.

Teacher Models: Large LLMs (Qwen3-32B, Gemma3-27B) are first initialized via fine-tuning on the WebNLG dataset (40k knowledge graph–text pairs). This provides broad, high-quality D2T supervision.
Synthetic Data Generation: The teacher model generates synthetic text descriptions for the limited target-domain structured inputs (100 instances per domain from the QUINTD-1 benchmark).
Student Training: A small student model is trained on these $(input, synthetic\_text)$ pairs using standard maximum-likelihood estimation.
Teacher Variants: The study compares distillation from a Zero-Shot (ZS) teacher versus a WebNLG-SFT teacher.

2. Structure-Preserving Augmentation

To overcome the scarcity of the 100 seed instances per domain, the authors introduce augmentation strategies that expand the training data while preserving the structural integrity of the inputs:

Structured Subsampling: Exploiting the compositional nature of structured inputs (e.g., entity-attribute-value pairs), the method generates new training instances by extracting non-empty subsets of attributes. This creates simpler, faithful versions of complex inputs.
Noise-Based Structural Perturbation: To prevent overfitting to specific instances, the method introduces controlled noise by randomly modifying values within instances (e.g., altering a specific temperature reading in a weather forecast) or swapping structural blocks across instances.
Mixed Strategy: The final training set combines original, subsampled, and perturbed instances to expose the student to diverse input distributions.

3. Baselines and Controls

The study compares DDKD against:

Zero-Shot (ZS): Direct prompting of large and small LLMs.
Source-Domain SFT: Fine-tuning small models directly on WebNLG data without target-domain distillation.
Real-Data Scaling (QUINTD-5): A control experiment where the number of real target-domain inputs is increased fivefold (from 100 to 500) without augmentation, to test if simply scaling real data yields better results than augmentation.

4. Evaluation

Since no reference text exists, evaluation is reference-free:

Faithfulness: An "LLM-as-a-Judge" protocol (using GPT-5.1 and Gemini-2.5-Pro) evaluates generated text against the structured input, categorizing errors into: Incorrect Fact, Not Checkable, Misleading, and Other.
Content Coverage: A sanity check ensures error reduction is not achieved by conservative under-generation (omitting content).
Human Validation: A subset of outputs is annotated by human experts to verify the reliability of the LLM judges.

Key Contributions

Investigation of DDKD for Cross-Domain D2T: The paper demonstrates that data-driven knowledge distillation is a viable strategy for generating text from structured data in domains where no training text exists, effectively transferring capabilities from source domains (WebNLG) to target domains.
Structure-Preserving Augmentation: The authors introduce and validate that augmenting target-domain inputs via structural subsampling and perturbation is more effective and cost-efficient than simply collecting more real data. This strategy decomposes complex generation tasks into simpler sub-problems for the teacher.
QUINTD-5 Benchmark: They construct QUINTD-5, a fivefold extension of the QUINTD-1 benchmark, to serve as a controlled baseline for comparing data augmentation against real-data scaling.
Rigorous Reference-Free Evaluation: The paper establishes a robust evaluation pipeline combining LLM-as-a-Judge with error taxonomies, inter-judge agreement analysis, and human validation, proving that small distilled models can achieve high faithfulness without sacrificing content coverage.

Results

Experiments were conducted on five benchmarks: Wikidata, Ice Hockey, OpenWeather, GSM Arena, and OWID (chart captioning).

Performance of DDKD: At a constant model size (1.7B parameters), DDKD consistently outperformed both zero-shot inference and source-domain fine-tuning across all five domains.
Small vs. Large Models: The distilled 1.7B models outperformed much larger fine-tuned models (32B parameters) on four of the five domains (Wikidata, Ice Hockey, GSM Arena, and OWID) and achieved comparable performance on the remaining one (OpenWeather).
Augmentation vs. Scaling: The DDKD approach using structure-preserving augmentation on 100 seed instances significantly outperformed the approach of scaling to 500 real instances (QUINTD-5). The authors attribute this to the efficiency of generating diverse, simplified examples that help the model learn structural patterns better than raw data scaling.
Teacher Selection: Distillation from a WebNLG-finetuned teacher generally yielded better results than from a zero-shot teacher, particularly in domains requiring structured graph-to-text verbalization (e.g., Wikidata). However, for domains where pre-trained knowledge is already strong (e.g., GSM Arena), zero-shot teachers were competitive.
Faithfulness and Coverage: The distilled models achieved lower error rates without reducing content coverage, confirming that performance gains were not due to conservative omission of information.

Significance and Claims

The paper claims that cross-domain generalization for data-to-text generation can be effectively transferred to compact models when large LLMs are leveraged as intermediate generators rather than final deployment targets.

Key takeaways include:

Feasibility without Target Data: It is possible to train effective small models for new D2T tasks using only structured inputs and synthetic supervision, eliminating the need for costly human-annotated reference text in the target domain.
Efficiency of Augmentation: Structure-preserving augmentation is a superior strategy to real-data scaling for distillation, offering higher data efficiency and better generalization to schema heterogeneity.
Practical Deployment: The approach enables the deployment of efficient, small-scale models (1.7B) that rival or exceed the performance of significantly larger models in specific cross-domain scenarios, addressing the computational bottlenecks of long-context inference and training.

The authors remain modest regarding the transferability of these gains to significantly larger architectures, noting their focus on small-scale models due to the computational constraints of long-context processing. They also acknowledge that the heuristic design of structural augmentation relies on inductive biases that may not be optimal for all conceivable domains.