Breaking the HISCO Barrier: Automatic Occupational Standardization with OccCANINE

1. The Problem

Historical research in economics, sociology, and history relies heavily on HISCO (Historical International Standard Classification of Occupations) codes to categorize occupational data from sources like census records, marriage certificates, and parish registers.

• The Bottleneck: Converting raw, unstructured textual descriptions of occupations into standardized HISCO codes is traditionally a manual, labor-intensive process.
• Scale: A single researcher might take 10 seconds to code one description. Coding 10,000 descriptions takes ~28 hours; millions of records make manual coding infeasible.
• Limitations of Previous Solutions:
  • Rule-based/String Matching: Traditional methods rely on cleaning strings (removing stop words, fixing typos) and using regular expressions. These are brittle, prone to errors due to spelling variations, and lack generalizability across different data sources.
  • Early Machine Learning: Previous ML approaches (e.g., bag-of-words, nearest neighbors) achieved moderate accuracy (65–89%) but often required high production rates (manual review of 20–40% of data) or extensive feature engineering.
  • Ambiguity: Occupational descriptions often contain multiple jobs, negations, or archaic spellings, which surface-level matching fails to interpret semantically.

2. Methodology: OccCANINE

The authors introduce OccCANINE, an open-source, transformer-based language model designed to map occupational descriptions to HISCO codes (and other systems) automatically.

A. Architecture

• Base Model: Built on CANINE (Clark et al., 2022), a 127-million-parameter encoder. CANINE was chosen for its character-level tokenization, which makes it robust to typos and spelling variations common in archival data (e.g., "frmter" vs. "farmer").
• Encoder-Decoder Structure:
  • Encoder: Takes raw text (and optionally a language identifier) and converts it into a 768-dimensional semantic embedding.
  • Decoders: Two distinct decoding strategies are implemented:
    1. Flat Decoder ("Fast"): A linear classification head that outputs probabilities for all ~1,919 HISCO codes simultaneously. Requires a confidence threshold tuning.
    2. Sequential Decoder ("Good"): A transformer decoder that predicts the HISCO code digit-by-digit (e.g., predicting '6', then '1', then '1'...). This approach handles ambiguity better, naturally manages multi-occupation outputs, and eliminates the need for explicit threshold tuning.

B. Training Data

• Scale: The model is fine-tuned on 15.8 million occupational description-code pairs.
• Sources: Data aggregated from 29 different sources (e.g., Danish, Swedish, UK, and US censuses; marriage records; ship logs) covering 13 languages.
• Augmentation:
  • Conjunction Augmentation: The authors artificially generated multi-occupation phrases (e.g., "farmer AND fisherman") by combining single-occupation strings to teach the model to output multiple codes.
  • Adversarial Training: Includes character/word insertions, back-translation, and random noise to improve robustness.
  • Order-Invariant Loss: A novel loss function was developed to handle cases where the order of multiple occupation codes in the training labels varies (e.g., [Farmer, Fisherman] vs. [Fisherman, Farmer]), ensuring the model learns the set of occupations rather than a specific sequence.

C. Training Procedure

• Parameters: Total model size is ~151 million parameters (Encoder + Decoder).
• Steps: Trained over 1.6 million steps with a batch size of 512.
• Regularization: Uses 10% dropout, random text augmentation, and occasional masking of the language label to force the model to learn language-independent semantic patterns.

3. Key Contributions

1. High-Accuracy Automation: Achieves 96.1% accuracy (5-digit level) on a held-out test set, significantly outperforming previous rule-based and ML methods.
2. Generalizability: The model generalizes across 13 languages and 29 diverse data sources without source-specific engineering.
3. Multi-System Support: Beyond HISCO, the authors demonstrate the method's portability by fine-tuning models for OCC1950 (US), OCCICEM (UK), and ISCO-68, achieving 85–97% accuracy.
4. Efficiency: Reduces coding time from weeks/months to minutes. On a standard laptop GPU, it processes 10,000 records in under 9 minutes (Sequential) or under 2 minutes (Flat).
5. Open Science: The code, models, and training pipelines are released as open-source on GitHub.

4. Key Results

• Performance Metrics (Sequential Decoder, 5-digit level):
  • Accuracy: 96.1%
  • Precision: 96.1%
  • Recall: 96.7%
  • F1-Score: 96.3%
• Out-of-Distribution (OOD) Performance:
  • The model maintains high accuracy (87–96%) on manually validated datasets it was not trained on (e.g., Copenhagen Burial Records, Norwegian Student Biographies).
  • Performance drops in datasets with low training representation (e.g., German, Italian) or where legacy regex-based coding introduced systematic errors (e.g., UK Bankruptcies), but remains competitive with human alternatives.
• Top-k Decoding:
  • When the model is allowed to provide a shortlist of candidates (Top-2), the probability of including the correct code jumps to >98%. This suggests many errors are "near misses" rather than complete failures, making the tool highly effective for assisted coding (human-in-the-loop).
• Socio-Economic Bias: Analysis shows a negligible correlation between model performance and the socio-economic status (SES) of the occupation, once controlling for the frequency of training examples.

5. Significance and Impact

• Democratization of Research: By removing the "HISCO barrier," OccCANINE allows researchers to process massive historical datasets that were previously too large to code manually. This enables new studies on social mobility, women's empowerment, and industrialization at a scale previously impossible.
• Methodological Shift: The paper moves the field from brittle, rule-based string matching to semantic understanding using deep learning. It proves that small, specialized language models can outperform larger general-purpose models in specific historical tasks.
• Scalability: The approach offers a "recipe" for standardizing other unstructured historical data, such as trade goods, educational backgrounds, or medical diagnoses.
• Practical Workflow: The authors provide a clear escalation strategy:
  1. Full Automation: Use for high-confidence cases.
  2. Assisted Coding: Use Top-k decoding for low-confidence cases to speed up human review.
  3. Fine-tuning: If a specific dataset performs poorly, the model can be fine-tuned on a small set (e.g., 10,000 labeled examples) to adapt to local coding conventions.

In conclusion, OccCANINE represents a paradigm shift in economic history, transforming occupational coding from a bottleneck into a scalable, automated process while maintaining high fidelity to historical nuance.