Benchmarking Classical and Transformer-Based Models for Document Sensitivity Classification
This paper introduces Strategic 16K, a leakage-controlled corpus of 16,000 diplomatic cables, to establish the first reproducible benchmark for document sensitivity classification that reveals how residual markers inflate model performance and demonstrates that BERT outperforms other architectures when trained on clean data.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the vast, quiet corridors of modern organizations, a single misplaced document can trigger a cascade of consequences, from regulatory fines to national security breaches. For decades, the task of deciding whether a file is safe to share or must be kept secret has fallen to human reviewers. These experts read through cables, memos, and reports, manually assigning labels like "Confidential" or "Secret." It is a slow, expensive process that struggles to keep pace with the sheer volume of information generated today. To solve this, scientists have turned to artificial intelligence, hoping to teach computers to read with the same discernment. However, a hidden trap has plagued these efforts: the training data itself often contains clues that distort the test. If a computer learns to spot the word "Secret" written in the middle of a sentence rather than understanding the actual meaning of the text, it will fail the moment it encounters a real document where that clue has been removed. This paper tackles that specific problem, asking whether machines can truly learn to recognize sensitive information or if they are simply memorizing the shortcuts left behind by human data preparation.
The researchers set out to build a fair test using a massive collection of real diplomatic cables released by WikiLeaks, known as the Public Library of US Diplomacy. This archive contains over 250,000 documents, each originally stamped with an official classification by government officials. The team first had to clean this data, stripping away the very markers that make the labels obvious. They removed single-letter codes like (S) for Secret that appeared at the start of paragraphs, deleted sentences that explicitly stated "this cable is classified," and erased repetitive distribution warnings. They called their final, purified dataset Strategic 16K, a collection of 16,000 documents where the only way to tell if a file is sensitive is to understand its actual content. This step was crucial; without it, a computer could achieve near-perfect scores simply by scanning for the word "Secret" rather than learning what makes a document sensitive.
With this clean dataset in hand, the team put six different types of computer models to the test. They compared older, simpler methods that count how often words appear against newer, more complex systems based on a technology called transformers, which are designed to understand the context and relationship between words in a sentence. The results showed a clear divide. The most advanced transformer models, specifically one called BERT, achieved the highest accuracy, correctly identifying sensitive documents about 89 percent of the time. Another transformer model, ELECTRA, came in a close second. These systems succeeded because they learned to look at the entire story of a document, understanding how different words work together to convey a secret, rather than just hunting for specific keywords.
The study also found that the older, simpler models were not entirely useless. A method based on counting word frequencies, when paired with a standard statistical algorithm, managed to get about 86 percent of the answers right. While this is lower than the top performers, it is a strong result for a system that requires far less computing power and time to run. The researchers noted that for organizations with limited resources, this simpler approach offers a practical balance between speed and reliability. However, the most important takeaway was the proof that the "distortion" had been stopped. When the researchers tested the models on the original, uncleaned data, the simple models scored nearly 99 percent, a number that was clearly inflated by the presence of the hidden markers. Once those markers were gone, the scores dropped to a more realistic level, proving that the models were now forced to learn the genuine signals of sensitivity.
The paper concludes that while the most powerful artificial intelligence models are currently the best at this task, the field must remain vigilant about how data is prepared. The researchers suggest that in the future, instead of relying on a single giant model, organizations might use a team of specialized systems working together. One system could look for specific names of people or places, another for the topic being discussed, and a third for the structure of the document. This approach would not only improve accuracy but also explain exactly why a document was flagged as sensitive, a feature that is vital for trust in security systems. For now, the creation of Strategic 16K stands as a new standard, offering a clear, honest benchmark that allows scientists to measure progress without the distortion of hidden clues.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.