💬 NLP

TradeVerse: A Longitudinal Benchmark of Political Negotiation in International Trade

This paper introduces TradeVerse, the first longitudinal benchmark derived from WTO meeting records that evaluates large language models' capabilities in understanding multi-turn political trade negotiations through three tasks: predicting product codes, identifying responding countries, and generating diplomatic statements.

Original authors: Debodeep Banerjee, Amitangshu Dasgupta

Published 2026-08-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Debodeep Banerjee, Amitangshu Dasgupta

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the high-stakes world of international trade, agreements are rarely signed in a single moment of clarity. Instead, they emerge from a slow, often contentious process of negotiation that can stretch across years. When one nation raises a concern about another's tariffs or regulations, the response is not a simple yes or no. It is a dialogue, a back-and-forth exchange where positions shift, alliances form, and arguments evolve over multiple meetings. For decades, these conversations have been recorded in the minutes of the World Trade Organization, creating a vast, longitudinal archive of real-world diplomacy. Until recently, the artificial intelligence systems designed to read and understand human language had only been tested on static documents or isolated questions. They had not been asked to navigate the complex, unfolding drama of a multi-year political dispute.

A new study introduces a tool designed to test whether modern artificial intelligence can truly understand this kind of long-term political reasoning. The researchers built a benchmark called TRADEVERSE, which reconstructs 1,170 specific trade concerns from the World Trade Organization's records. These concerns span more than three decades, involving 6,933 separate meetings and 26,219 individual statements. The dataset covers five different committees and 89 distinct product categories, ranging from agricultural goods to electronics. The goal was not to see if a computer could simply summarize a text, but to see if it could follow the thread of a conversation that might have started five years ago, remember who said what, and understand the strategic position of each country involved.

The researchers set up three distinct challenges to test the intelligence of six leading language models. The first task asked the models to look at the entire history of a trade dispute and identify which products were being discussed. In the real world, trade concerns are often classified by a specific system of codes that categorize goods. The models had to predict these categories based solely on the text of the negotiations. The results showed that while the models were good at finding the right products, they were also prone to guessing too many possibilities. They tended to list a wide range of related items rather than pinpointing the exact ones, suggesting they were hedging their bets when the evidence was not perfectly clear.

The second challenge was a test of anonymity and inference. The researchers took the meeting records and removed the names of the countries involved, replacing them with generic placeholders. The models were then asked to guess which country was responding to the complaints. This was designed to see if the AI could figure out the answer based on the substance of the argument—the specific laws, the nature of the products, and the style of the defense—rather than just recognizing a country's name. The models performed with high accuracy, correctly identifying the responding country in most cases. However, a significant pattern emerged: the models were much better at identifying Western nations than non-Western ones. This gap in performance persisted even when all country names were hidden, indicating that the AI's training data contained more examples of Western diplomatic language, making it easier for the system to recognize those patterns.

The final and perhaps most difficult task was to act as a participant in the negotiation. The models were given the history of a dispute and the statements made by the complaining countries in the final round. Their job was to write the concluding statement for the responding country. The researchers compared these generated statements against the actual historical records. While the AI produced text that sounded fluent and diplomatically appropriate, it often lacked the specific details found in the real-world responses. The models could mimic the tone of a diplomat but struggled to replicate the precise, substantive arguments that real nations use to defend their policies. Interestingly, the study found that the more rounds of negotiation a dispute had, the better the models performed at generating the final statement. This suggests that the longer the conversation history, the more context the AI had to understand the role it was playing.

The study concludes that while current artificial intelligence systems are powerful tools for reading text, they still face significant hurdles when asked to reason through the long, evolving dynamics of real-world political negotiations. They can identify patterns and mimic styles, but they often miss the specific strategic nuances that define international trade disputes. The researchers made their dataset and the tools they used available to the public, hoping that this new benchmark will help future systems learn to navigate the complex, longitudinal nature of human diplomacy. The work highlights that understanding a conversation is not just about reading the words on the page, but about tracking the history, the relationships, and the shifting strategies that unfold over time.

Technical Summary: TRADEVERSE

Problem Statement

Current benchmarks for Large Language Models (LLMs) in legal and political domains predominantly evaluate performance on static, isolated documents or single-turn decision-making tasks. However, real-world institutional decision-making, particularly in international trade, is inherently longitudinal. Negotiations evolve over multiple iterations, often spanning years, where the outcome of any specific turn is contingent upon the arguments, alliances, and stances established in previous rounds. Existing benchmarks fail to capture this dynamic, requiring models to track the chronology of multi-party dialogues, understand shifting geopolitical positions, and reason over the strategic structure of negotiations rather than just surface-level text comprehension.

Methodology

The authors introduce TRADEVERSE, a benchmark constructed from the World Trade Organization (WTO) Trade Concerns Database. The dataset comprises 1,170 specific trade concerns reconstructed from 6,933 meeting minutes spanning over three decades (1995–2026) across five WTO committees and 89 Harmonized System (HS) product chapters.

The dataset is structured as a sequence of negotiation rounds ( $r_{i,1}, \dots, r_{i,R_i}$ ) involving raiser states, supporter states, and respondent states. Unlike previous datasets that rely on manual annotation, TRADEVERSE derives all ground truth labels directly from official proceedings. The benchmark defines three distinct tasks to evaluate LLM capabilities:

Task 1: HS Chapter Prediction (Multi-label Classification):
Given the full longitudinal history of a trade concern, the model must predict the set of HS chapters (product categories) under discussion. This tests the model's ability to synthesize technical details scattered across multiple meetings. Note: Due to HS code availability constraints (present only in TBT and SPS groups), this task was evaluated on 724 of the 1,170 meetings.
Task 2: Respondent Identification (Anonymized Classification):
The model must identify the responding country based on the negotiation history. This is tested under two anonymization settings:
- ST-1: Only the respondent's name and demonyms are masked.
- ST-2: All country names, demonyms, and national agencies are masked.
  This evaluates whether the model can infer the respondent from the substance of the dispute (measures, products, legal arguments) rather than relying on explicit entity mentions.
Task 3: Final Statement Generation (Natural Language Generation):
Given the negotiation history up to the final round (including the raiser's and supporters' final statements), the model must generate the concluding statement of the respondent country. The generated text is evaluated against the actual historical statement.

Experimental Setup:
Six state-of-the-art LLMs (including Nemotron3-Ultra:550B, Llama-3.3:70B, GLM-5.2, DeepSeek-V4-Pro, GPT-OSS-120B, and Kimi-K2.7-Code) were evaluated in a zero-shot setting. Metrics included Precision, Recall, and F1 for classification tasks, and BLEU-4, ROUGE-L, and BERTScore for generation tasks.

Key Results

Task 1 (HS Prediction): Models demonstrated high recall but low precision, indicating a tendency to "hedge" by over-generating plausible product categories rather than committing to a precise set. For instance, GLM-5.2 achieved a Micro-F1 of 65.63, while DeepSeek-V4-Pro achieved 63.07.
Task 2 (Respondent Identification): Models achieved high overall accuracy (ranging from ~80% to ~93%). Crucially, performance remained stable between ST-1 and ST-2 settings, suggesting models infer the respondent from the dispute's substance rather than relying on explicit country names.
- Geopolitical Bias: A significant disparity was observed. All six models identified Western respondents with significantly higher accuracy than non-Western respondents (e.g., GPT-OSS-120B: 92.70% vs. 77.46%; $p < 0.01$ ). This gap persisted even when all identities were masked, suggesting the bias is rooted in the models' internal knowledge or training data distribution rather than surface cues.
Task 3 (Statement Generation): Generated statements were generally fluent and diplomatically appropriate but lacked specific content alignment with the actual historical interventions. GLM-5.2 outperformed other models in both lexical and semantic similarity (BERTScore-F1: 60.49).
- Round Correlation: A positive Spearman correlation was found between the number of negotiation rounds and generation quality (BERTScore), indicating that longer historical contexts provided better training signals for the models to adopt the respondent's persona.

Key Contributions

Longitudinal Benchmark: TRADEVERSE is the first benchmark to systematically evaluate LLMs on authentic, multi-party, longitudinal political trade negotiations, moving beyond static document understanding.
Annotation-Free Construction: The dataset reconstructs 1,170 concerns from 6,933 meeting records with ground truths extracted directly from official WTO proceedings, eliminating the need for manual human annotation.
Structured Reasoning Tasks: The benchmark defines three complementary tasks (HS prediction, anonymized respondent identification, and role-playing generation) that require integrating long-context understanding, institutional reasoning, and strategic language generation.
Empirical Findings on Bias: The study provides empirical evidence of systematic geopolitical biases in current LLMs, specifically a performance disparity in identifying Western versus non-Western actors in trade disputes, which persists under full anonymization.

Significance and Claims

The paper claims that TRADEVERSE offers a novel setting to study LLM reasoning under evolving institutional processes and geopolitical biases. The authors assert that current models, while capable of high-level fluency and basic entity inference, struggle with the specificities of realpolitik: they over-predict product categories, lack specificity in diplomatic responses, and exhibit stable biases favoring Western members.

The authors position TRADEVERSE as a tool for both the machine learning community (to improve negotiation and long-context reasoning) and social scientists (to analyze the evolution of disputes, coalition formation, and diplomatic language patterns at a scale previously unattainable through manual effort). The paper concludes by releasing the dataset, extraction pipeline, and evaluation code to facilitate further research into longitudinal institutional reasoning.