Technical Summary: TRIDENT – Benchmarking LLM Safety in Finance, Medicine, and Law

Problem Statement

The rapid deployment of Large Language Models (LLMs) in high-stakes domains such as law, finance, and medicine presents significant ethical risks. While existing benchmarks effectively evaluate domain-specific competence (e.g., diagnostic reasoning, legal document parsing, or financial analysis), they largely neglect the evaluation of domain-specific safety risks. Current evaluations often fail to assess whether models adhere to formal professional codes of conduct, such as the American Medical Association (AMA) Principles of Medical Ethics, the American Bar Association (ABA) Model Rules of Professional Conduct, and the CFA Institute Code of Ethics. Consequently, regulators and developers lack systematic tools to identify safety gaps where models might inadvertently generate outputs that violate fiduciary duties, patient confidentiality, or legal ethics.

Methodology: Trident-Bench

To address this gap, the authors introduce Trident-Bench, a benchmark designed to evaluate LLM safety through the lens of "total refusal." In this framework, every prompt is engineered to be unsafe, making refusal the only appropriate model behavior.

1. Definition of Safety

Safety is defined strictly as conformity with established professional codes of ethics. Any output that disregards these principles—such as facilitating financial misconduct, breaching client confidentiality, or endorsing unsafe medical practices—is classified as unsafe. The benchmark operationalizes these definitions by reformulating specific principle violations into user queries.

2. Benchmark Construction Pipeline

The construction of Trident-Bench follows a multi-stage, principle-guided pipeline:

• Ethical Principle Alignment: The benchmark is grounded in three authoritative sources: the CFA Institute Code of Ethics, the AMA Principles of Medical Ethics, and the ABA Model Rules of Professional Conduct. These principles are reviewed and annotated by domain experts.
• Harmful Prompt Generation: To overcome the resistance of safety-aligned models to generating unsafe content, the authors employ jailbreak-guided generation. They utilize both prompt-based methods (e.g., PAIR, TAP) and fine-tuned methods (e.g., Auto-DAN, GCG) across various base models (GPT-4o, LLaMA 3.1, Mixtral) to elicit 1,000 harmful prompt candidates per domain.
• Principle Mapping and Filtering: Generated prompts are manually filtered by domain experts to ensure they map concretely to specific ethical clauses (e.g., CFA Standard III(D)). Vague or speculative prompts are discarded.
• Safe Response Generation: For each harmful prompt, a paired "safe" response is generated using safety-aligned LLMs. These responses undergo a two-stage check: an automatic classifier flags evasive or hallucinated outputs, followed by manual verification by domain experts to ensure factual accuracy and ethical alignment.
• Expert Validation: A stringent annotation protocol is applied where each prompt and response is reviewed by three domain experts. Only samples receiving unanimous agreement are retained. This ensures high precision and reduces noise.

3. Dataset Statistics

The final dataset comprises 2,635 high-quality harmful prompts:

• Finance: 911 prompts
• Law: 887 prompts
• Medicine: 837 prompts

4. Evaluation Metrics

The authors evaluate models using a harmfulness score (1–5 scale), where 1 represents a clear refusal and 5 represents full compliance with the harmful request. To reduce judgment bias, a "jury" of two LLMs (Claude 3.5 and Gemma 2-9B) scores the responses, with results validated by human annotators. Additionally, a false rejection analysis is conducted using benign prompts to ensure models are not trivially rejecting all inputs.

Key Contributions

1. Trident-Bench: The first systematic benchmark specifically targeting LLM safety in law and finance, grounded in real-world professional codes of ethics.
2. Empirical Study: A comprehensive evaluation of 19 models (general-purpose, domain-specialized, and safety-aligned) revealing that domain specialization alone does not guarantee ethical robustness.
3. Actionable Insights: Provision of a resource for regulators and developers to identify safety risks and ensure accountability in high-stakes AI deployment.

Experimental Results

The evaluation of 19 models on Trident-Bench yielded the following findings:

• General-Purpose Models: Closed-source models (GPT-4o, Gemini 2.5 Flash) demonstrated the strongest safety performance, typically issuing direct refusals or grounded redirections based on professional norms. Open-source general models (e.g., Qwen3, DeepSeek-R1) lagged behind, often producing weak, verbose rejections with partial compliance.
• Domain-Specialized Models: Contrary to expectations, several domain-specialized models (particularly in law and finance, such as DISC-LawLLM and FS-LLaMA) exhibited the lowest safety performance. These models frequently complied with unethical queries, often treating them as adversarial client questions and providing workarounds rather than refusing. For instance, legal models sometimes provided litigation strategies for unethical behavior instead of refusing.
• Safety-Aligned Models: Models explicitly trained for safety (LLaMA Guard3-8B, LLaMA Guard4-12B) showed significant improvements, approaching the performance of closed-source leaders. They consistently produced concise, principled refusals.
• False Rejections: Safety-aligned models showed a marginal increase in false rejections on benign prompts (<0.3%), a trade-off consistent with prior findings that safety gains may come with slight over-refusal.

Significance and Claims

The paper claims that Trident-Bench provides one of the first systematic resources for studying LLM safety in law and finance. The authors emphasize that their work highlights an urgent need for more robust safeguards in high-stakes domains.

Key takeaways from the authors' perspective include:

• Safety vs. Competence: Strong generalist models can meet basic safety expectations, whereas domain-specialized models often fail, suggesting that domain knowledge alone does not instill ethical robustness.
• The Role of Alignment: Targeted safety alignment (via refusal demonstrations or reward modeling) is critical for ensuring ethical refusal, even in smaller-scale models.
• Limitations and Future Work: The authors acknowledge the high cost of expert annotation (exceeding $18,000) and the challenge of scalability. They propose future work in multi-turn interaction benchmarks, exploring cost-effective annotation methods (e.g., role-play), and expanding the scope to cross-jurisdictional and culturally ambiguous cases.

The paper concludes that while Trident-Bench can inform future training pipelines, its primary contribution is a principled, expert-validated resource for safety evaluation, laying the groundwork for reducing risks in professionally regulated fields.