Build, Borrow, or Just Fine-Tune? A Political Scientist's Guide to Choosing NLP Models

This paper guides political scientists in choosing NLP strategies by demonstrating that fine-tuning general models like ModernBERT often suffices for high-frequency tasks, reserving the need for specialized, domain-specific models for rare event categories where performance gaps are most pronounced.

The Gist

Imagine you are a political scientist trying to solve a mystery: Who is doing what, where, and how in the world of global conflict? You have a massive library of news reports (the Global Terrorism Database) and you need a robot assistant to read them and sort every incident into categories like "Bombing," "Kidnapping," or "Assassination."

The big question this paper asks is: How do you build that robot?

The author, Shreyas Meher, presents three ways to get your robot ready, using a "Build, Borrow, or Buy" framework. Here is the breakdown in plain English with some creative analogies.

The Three Options

1. Build (The "Master Chef" Approach)

• What it is: You start from scratch. You gather millions of pages of specific conflict news, teach the robot the language of war from the ground up, and train it for weeks on powerful computers.
• The Analogy: This is like hiring a master chef, buying a farm, growing your own vegetables, and teaching them your family's secret recipes from birth.
• Pros: The robot knows the "dialect" of conflict perfectly. It understands subtle differences between a "barricade incident" and a "kidnapping."
• Cons: It is incredibly expensive, takes months, and requires a team of experts. It's like building a custom Ferrari when you just need a car to get to the grocery store.

2. Borrow & Fine-Tune (The "Apprentice" Approach)

• What it is: You take a robot that was already trained on everything (the whole internet, Wikipedia, news, books) and give it a crash course on your specific conflict data.
• The Analogy: This is like hiring a brilliant, well-traveled chef who knows how to cook Italian, French, and Chinese food. You don't teach them everything; you just give them a few days of training on your specific family recipes. They already know how to chop, sauté, and season; they just need to learn your specific dish.
• Pros: It's fast (a weekend), cheap (a few dollars in cloud computing), and easy.
• Cons: They might miss the tiny, obscure details that the "Master Chef" knows.

3. Buy (The "Takeout" Approach)

• What it is: You don't train a robot at all. You just send your text to a giant commercial AI (like a super-smart chatbot) and ask, "What category is this?"
• The Analogy: This is like ordering takeout. You don't cook; you just pay a fee, and someone else hands you the food.
• Pros: Instant. No cooking required.
• Cons: It's often wrong on specific details, you can't control how they made it, and if the restaurant closes or changes the menu, your research breaks. Plus, it gets expensive if you order a lot.

---

The Experiment: The "Taste Test"

The author decided to test these approaches.

• The "Master Chef" (ConfliBERT): A robot specifically trained on conflict data (the current gold standard).
• The "Apprentice" (Confli-mBERT): A modern, general-purpose robot (ModernBERT) that was fine-tuned on the same conflict data.
• The "Takeout" (Commercial APIs): Various big AI models asked to guess the categories without any training.

The Results:

1. The "Common" Cases (The Main Course):
   For the most common events—like Bombings and Armed Assaults (which make up 98% of the data)—the "Apprentice" and the "Master Chef" were almost identical.

   • Analogy: If you ask both chefs to make a standard cheeseburger, they both make a perfect one. The "Master Chef" didn't add any extra magic. The "Apprentice" was just as good.

2. The "Rare" Cases (The Exotic Ingredients):
   For very rare events—like Hijackings or Barricade Incidents (less than 2% of the data)—the "Master Chef" was noticeably better.

   • Analogy: If you ask them to make a dish using a rare, obscure spice that only appears once in a thousand cookbooks, the "Master Chef" (who studied that specific spice for years) gets it right. The "Apprentice" guesses, and gets it wrong more often.

3. The "Takeout" (Commercial APIs):
   The commercial chatbots were disappointing. Even the smartest ones got the basic categories wrong more often than the trained robots.

   • Analogy: Ordering a custom dish from a fast-food chain. They might guess "burger," but they'll likely mess up the specific ingredients you asked for.

---

The Big Lesson: The Decision Framework

The paper concludes that you don't always need the "Master Chef." Here is the simple rule for political scientists (and anyone doing data work):

1. Look at your "Menu" (Prevalence):
If your research is about common things (like bombings), Fine-Tuning (The Apprentice) is perfect. It's cheap, fast, and just as accurate as the expensive custom model.
If your research is about rare, obscure things (like specific types of hijackings), you might need the Domain-Specific Model (The Master Chef) to get the details right.

2. Check your "Budget" (Resources):
Building a custom model costs thousands of dollars and months of time. Fine-tuning costs a few dollars and a weekend. Unless you really need those rare details, the "Apprentice" is the smarter financial choice.

3. Avoid "Takeout" (Commercial APIs) for serious work:
Unless you are just doing a quick, rough guess, don't rely on commercial AI APIs. They are less accurate, cost money every time you use them, and you can't save the "recipe" for later. If you need to reproduce your results in five years, the API might not even exist anymore.

The Bottom Line

Don't build a Ferrari if you just need to drive to the store.

For most political science research, the "Apprentice" approach (taking a smart, general AI and giving it a quick, specific training) is the sweet spot. It's accessible, affordable, and accurate enough for 98% of the job. You only need to invest in the expensive, custom "Master Chef" model if your research depends entirely on finding those tiny, rare needles in the haystack.

The future of AI in science isn't about building bigger, more expensive models; it's about using the smart tools we already have and teaching them just enough to do the specific job.

Technical Summary

Here is a detailed technical summary of the preprint "Build, Borrow, or Just Fine-Tune? A Political Scientist's Guide to Choosing NLP Models" by Shreyas Meher.

1. Problem Statement

Political scientists increasingly rely on Natural Language Processing (NLP) for tasks like conflict event classification, sentiment analysis, and text coding. However, researchers face a methodological dilemma regarding model selection:

• Build: Pretrain a domain-specific model from scratch (e.g., ConfliBERT). This offers high domain specificity but requires massive computational resources, curated corpora, and deep expertise.
• Borrow/Fine-Tune: Adapt a general-purpose transformer (e.g., ModernBERT) using labeled task data. This is accessible and cost-effective but relies on the assumption that general representations are sufficient.
• Buy: Use commercial Large Language Model (LLM) APIs via zero-shot prompting. This requires no training but lacks reproducibility, control, and often accuracy.

The paper addresses the lack of empirical guidance on the marginal value of domain-specific pretraining versus fine-tuning general models, specifically asking: Is the performance gain of a specialized model worth the significant increase in cost and complexity for typical political science research questions?

2. Methodology

The study uses Conflict Event Classification on the Global Terrorism Database (GTD) as a test case. The GTD contains over 200,000 events classified into nine attack types (e.g., Bombing, Armed Assault, Kidnapping), presenting a multi-label classification problem with severe class imbalance (the most frequent class is 95x larger than the rarest).

Experimental Setup

• Models Compared:
  1. Confli-mBERT: A model created by the author by fine-tuning ModernBERT-base (149M parameters, trained on 2 trillion tokens) on GTD data.
  2. ConfliBERT: A domain-specific pretrained model (the current gold standard) pretrained on 33 million tokens of conflict-specific text.
  3. ConflLlama: A fine-tuned generative model (Llama-based) for comparison.
  4. Zero-Shot Baselines: Six commercial and open-source LLMs (e.g., Gemini, Claude, Qwen) tested via API and local deployment without fine-tuning.
• Data Split: A temporal split was used to ensure realistic evaluation: training on pre-2017 data ($N=170,623$) and testing on 2017+ data ($N=37,709$).
• Architecture & Training:
  • Head: Multi-label classification using Sigmoid activation and Binary Cross-Entropy with Logits (BCEWithLogitsLoss).
  • Imbalance Handling: Applied inverse-frequency class weighting to the loss function. The rarest class (Hijacking) received ~136x the weight of the most common class (Bombing) to prevent the model from ignoring rare events.
  • Hardware: Fine-tuning was performed on a single NVIDIA A100 GPU (~4 hours).
• Evaluation Metrics: Overall Accuracy, Per-class F1, Micro/Macro F1, AUC-ROC, and True Positive counts.

3. Key Results

A. Fine-Tuning vs. Domain-Specific Pretraining

• Overall Accuracy: Confli-mBERT achieved 75.46%, compared to ConfliBERT's 79.34%. While ConfliBERT is superior, the gap is modest (~4 percentage points).
• The Prevalence Effect: The performance gap is not uniform. It is directly correlated with class prevalence.
  • High-Prevalence Classes: For common events (Bombing, Armed Assault, Kidnapping), the models are nearly indistinguishable. For example, on Bombing/Explosion, F1 scores are 0.95 (Confli-mBERT) vs. 0.96 (ConfliBERT).
  • Low-Prevalence Classes: The gap widens significantly for rare events (<2% of data). ConfliBERT outperforms Confli-mBERT by large margins on Hijacking, Barricade Incidents, and Unarmed Assault (e.g., Hijacking F1: 0.70 vs. 0.37).
• True Positives: Across the entire test set, ConfliBERT identified only 265 more true positives (0.8% increase) than Confli-mBERT. For common categories, Confli-mBERT actually captured more true instances in some cases (e.g., Armed Assault).
• Conclusion: Domain-specific pretraining acts as an "informative prior" that is crucial only when labeled data is scarce. For common categories with thousands of examples, fine-tuning a modern general model is sufficient.

B. Fine-Tuning vs. Zero-Shot LLMs (The "Buy" Option)

• Performance Gap: Zero-shot models performed significantly worse. The best commercial API (Gemini 3 Flash) achieved 65.85% accuracy, while the best fine-tuned model (Confli-mBERT) achieved 75.80%.
• Size vs. Performance: There was a negative correlation between model size and classification performance in zero-shot settings. A small, fine-tuned 110M parameter model (ConfliBERT) outperformed a 685B parameter model (DeepSeek V3) by a wide margin.
• Cost & Reproducibility:
  • Cost: Fine-tuning cost $5–$15 in cloud compute. Classifying the full GTD via APIs would cost $33–$150 for a single pass, excluding development iterations.
  • Reproducibility: API models are black boxes with version instability. Fine-tuned models are static, reproducible, and can run locally on consumer hardware.

4. Key Contributions

1. Empirical Evidence on Marginal Returns: The paper provides concrete data showing that the "diminishing returns" curve of domain-specific pretraining is steep. The bulk of performance gains come from fine-tuning; the additional gain from full domain pretraining is concentrated almost entirely in rare event categories.
2. Decision Framework: The author proposes a practical framework for political scientists based on three factors:
   • Class Prevalence: If the research focuses on common events, fine-tune. If it focuses on rare events, consider domain-specific models.
   • Error Tolerance: If the analysis is aggregate (e.g., national trends), fine-tuning is robust enough. If individual event coding is critical, specialized models or human verification are needed.
   • Resources: Fine-tuning is accessible to individual researchers; building domain models requires institutional-level resources.
3. Refutation of "Buy" as a Primary Strategy: The study demonstrates that for structured classification tasks with available labeled data, "buying" (APIs) is inferior in accuracy, cost, and reproducibility compared to "borrowing" (fine-tuning).

5. Significance and Implications

• Democratization of NLP: The findings suggest that political scientists do not need to build expensive domain-specific models to achieve state-of-the-art results for most research questions. Fine-tuning modern general-purpose transformers (like ModernBERT) is a scientifically defensible, low-cost, and high-performance strategy.
• Shifting Baselines: As general-purpose models are trained on larger corpora (e.g., 2 trillion tokens vs. 3 billion), the "vocabulary gap" between general and domain text narrows. This implies that the necessity for domain-specific pretraining will decrease over time, further favoring the fine-tuning approach.
• Methodological Guidance: The paper moves the discipline away from the assumption that "more specialized = better" toward a nuanced view where task-data alignment and resource constraints dictate model choice.

Final Recommendation: For the majority of political science applications involving common event types, researchers should fine-tune the best available general-purpose model. Domain-specific models should be reserved for research questions heavily reliant on rare event detection or where the specific domain knowledge cannot be captured by general pretraining.