JFinTEB: Japanese Financial Text Embedding Benchmark

1. Problem Statement

Despite Japan's status as a major global financial market and the increasing adoption of NLP technologies within its financial institutions, there is a critical lack of standardized benchmarks for evaluating Japanese financial text embeddings.

• Gap in Existing Resources: Current benchmarks like MTEB (English), MMTEB (Multilingual), and JMTEB (General Japanese) lack financial domain specificity. Conversely, domain-specific benchmarks like FinMTEB cover English and Chinese but exclude Japanese.
• Domain Specificity: Japanese financial texts (e.g., quarterly reports, securities filings, regulatory disclosures) contain highly specialized terminology, formulaic phrasing, and cultural contexts that general-purpose corpora do not capture.
• Limitations of Current Approaches: Existing evaluations often fail to address specific financial needs such as regulatory document analysis, market sentiment understanding, and industry classification, leading to suboptimal model selection for real-world financial applications.

2. Methodology

The authors introduced JFinTEB, a comprehensive benchmark designed to evaluate text embeddings specifically for the Japanese financial domain.

A. Task Design and Dataset Construction

The benchmark comprises 11 validated tasks across three categories, utilizing both existing datasets and newly constructed ones:

1. Classification Tasks (6 tasks):
   • Sentiment: chABSA (aspect-based), Sentiment, and Horizon (future outlook vs. current state from Economic Watchers Survey).
   • Domain/Industry: Domain classification, Wikinews (politics vs. economics), and two new datasets (Industry 17 and Industry 33) aligning Japanese Wikipedia company descriptions with official JPX industry sectors at coarse and fine granularities.
2. Retrieval Tasks (4 tasks):
   • Instruction-following: JaFIn (financial FAQs) and PFMT (multi-turn regulatory Q&A).
   • Document Retrieval: Wikinews (headline-to-article) and Wikipedia (company name-to-description), constructed via automated matching without manual annotation.
3. Clustering Tasks (1 task):
   • Reasoning: "Reason" task grouping Economic Watchers Survey comments into 13 economic reasoning categories.

B. Quality Assurance

To ensure reliability, the authors applied two stability criteria:

• Model Family Consistency: Tasks were excluded if they showed performance reversals across different model families (e.g., small vs. large parameters).
• Validation-Test Stability: Tasks with >20% score divergence between validation and test sets were excluded.
• Result: Two initial datasets (MultiFin-ja and Industry 33) were excluded, leaving 11 stable tasks.

C. Evaluation Protocol

• Metrics: Macro-F1 for classification, NDCG@10 for retrieval, and V-measure for clustering.
• Models Evaluated: 14 representative embedding models were tested, including:
  • Japanese-Specialized: Sarashina (1.2B), Ruri v3 series (30M–310M), GLuCoSE.
  • Multilingual: Jina, Multilingual E5 (Small/Base/Large), OpenAI text-embedding-3 (Small/Large).
  • Domain Adaptation Baselines: BERT and FinBERT (financially pre-trained).

3. Key Contributions

1. First Comprehensive Benchmark: JFinTEB is the first unified benchmark specifically for Japanese financial text embeddings, filling a significant gap in the NLP community.
2. Diverse Task Coverage: It covers realistic financial scenarios (regulatory Q&A, market sentiment, industry classification) across classification, retrieval, and clustering.
3. Systematic Evaluation: Provides baseline results for 14 models, offering practical insights for model selection based on parameter size and architecture.
4. Open Resource: All datasets, evaluation code, and baseline results are publicly released to facilitate reproducible research.

4. Key Results

The evaluation of 14 models yielded several critical insights:

• Japanese vs. Multilingual Models:
  • Japanese-specialized models (Sarashina, Ruri) achieved slightly higher average scores (80.9 and 79.4, respectively) compared to multilingual models (E5-large, Jina, OpenAI at ~79.0–79.3).
  • However, the gap is narrowing, indicating significant improvements in the Japanese capabilities of multilingual models.
• Task-Specific Performance:
  • Classification: Sarashina ranked first (81.7 avg), followed by E5-large.
  • Retrieval: Jina ranked first (94.0 avg), closely followed by Sarashina.
  • Robustness: Sarashina demonstrated the most consistent high performance across both classification and retrieval categories.
• Model Size Correlation: Larger models consistently outperformed smaller ones across all families.
  • >1B parameters: Sarashina is the optimal choice.
  • 100M–1B parameters: Ruri 310M is a strong candidate.
  • <100M parameters: OpenAI (via API) or Ruri 30M are preferred.
• Domain Adaptation: FinBERT outperformed general BERT, confirming that domain-specific pre-training benefits Japanese financial tasks, similar to findings in English and Chinese.
• Retrieval Saturation: Several retrieval tasks showed high performance scores, suggesting that current embedding models are already mature for well-defined financial information needs, though harder tasks (long documents, cross-document reasoning) are needed for future stress-testing.

5. Significance and Future Work

• Industry Impact: JFinTEB provides a standardized protocol for financial institutions in Japan to evaluate and deploy embedding models for search, recommendation, and compliance systems.
• Research Foundation: It establishes a baseline for studying domain adaptation and cross-lingual transfer in a major non-English financial market.
• Limitations & Future Directions:
  • Current tasks focus on relatively short texts; future work will incorporate long-document scenarios (e.g., full financial reports).
  • Future iterations will include generative/RAG-based tasks and human-centered evaluations.
  • The benchmark will be maintained to incorporate new tasks as Japanese financial NLP applications evolve.

In conclusion, JFinTEB addresses a critical resource gap, enabling the systematic advancement of Japanese financial text mining and providing a robust foundation for both academic research and industrial application.