Technical Summary: Rethinking LLMOps for Fraud and AML

1. Problem Statement

The deployment of Large Language Models (LLMs) in fraud detection and Anti-Money Laundering (AML) compliance faces unique challenges that generic chat-oriented serving stacks fail to address. Unlike open-ended conversational workloads, compliance inference is characterized by:

• Prefix-Dominated Prompts: Requests typically consist of a large, reusable shared block (policy instructions, risk taxonomies, schema definitions) followed by variable case-specific evidence.
• Structured, Short Outputs: The goal is often a compact JSON object, a list of risk labels, or a short narrative, rather than long-form generation.
• Multi-Tenancy and Strict Constraints: Workloads involve multiple task variants (e.g., classification, extraction, translation) on a single base model, requiring strict adherence to data locality, auditability, and output validity.

Default serving configurations often result in poor performance (high latency, low GPU utilization) because they are not tuned to these specific workload shapes. Furthermore, regulated environments require self-hosted solutions to avoid data-governance risks associated with external APIs, necessitating a compliant, open-weight model stack.

2. Methodology

The paper proposes a workload-aware LLMOps stack centered on self-hosted open-weight models (Meta Llama, Alibaba Qwen) and optimized inference runtimes (vLLM, SGLang). The methodology focuses on mapping specific optimization techniques to the distinct phenotypes of fraud/AML workloads.

Key Technical Components

• Runtime Tuning: Adjusting max num seqs, max num batched tokens, and KV-cache budgets to prevent split prefills and preemption.
• Memory Management: Utilizing PagedAttention to reduce fragmentation and Automatic Prefix Caching (APC) to reuse KV-cache blocks for shared policy prefixes.
• Multi-Adapter Serving: Implementing adapter- and prompt-length-aware batching. Instead of naive sequential processing, the system groups requests by LoRA adapter ID and then buckets them by input length to minimize padding and scheduling inefficiencies.
• Lifecycle Management: Introducing sleep/wake lifecycle control for sequential model pipelines, allowing inactive models to release VRAM while maintaining a faster wake path than a full cold start.
• Decode Optimization: Using speculative decoding (e.g., EAGLE-style) and strict stop tokens to reduce unnecessary generation overhead for short JSON outputs.
• Quality Assurance: A LLM-as-Judge framework acting as a pre-production quality gate. This layer combines deterministic schema checks, reference metrics (BLEU, ROUGE, cosine similarity), and multi-judge rubric scoring against expert-adjudicated calibration data to ensure compliance before rollout.

Reproducibility Track

To ensure reproducibility without exposing proprietary financial data, the authors constructed a benchmark using public synthetic datasets (IBM AML and SAML-D). These datasets were transformed into prefix-heavy compliance prompts containing reusable policy text, transaction evidence, and schema-constrained outputs. No proprietary customer records or internal investigation notes were used.

3. Key Contributions

The paper makes four primary contributions:

1. Workload Characterization: It defines fraud and compliance inference as a distinct workload class defined by long shared prefixes, structured short outputs, and frequent multi-tenant usage.
2. Reusable Serving Stack Design: It presents a concrete architecture that maps optimization techniques (APC, PagedAttention, multi-adapter batching) directly to AML/Fraud workload requirements.
3. Public Reproducibility Track: It establishes a methodology for evaluating serving stacks using synthetic public data (IBM AML, SAML-D), avoiding the disclosure of sensitive institutional data while maintaining realistic workload shapes.
4. Quality Gate Framework: It describes an LLM-as-Judge evaluation system that integrates deterministic checks and human-aligned scoring to validate serving configurations before deployment.

4. Results

Experiments were conducted on public-synthetic AML workloads and controlled serving benchmarks. The results demonstrate significant efficiency gains through workload-aware tuning:

• Throughput: Increased from 612–650 to 3,600 requests/hour (approx. 5.5–5.9× improvement) in long-context extraction workloads.
• Latency: Reduced P99 latency from 31–38 seconds to 6.4–8.7 seconds.
• Resource Utilization: GPU utilization improved from 12% to 78%, reducing normalized costs (GPU-hours per 1k successful requests) by approximately 83%.
• Multi-Adapter Efficiency: Adapter- and prompt-length-aware batching improved effective throughput by 3.90× compared to standard PEFT-style adapter loading.
• Lifecycle Management: Sleep/wake control reduced model availability time by 29–37× compared to cold starts for sequential pipelines.
• Speculative Decoding: In guided-JSON benchmarks, a single H100 with speculative decoding matched or exceeded the throughput of a 2×H100 baseline while reducing normalized costs by ~53%.

5. Significance and Claims

The paper argues that for regulated inference, performance is as much a workload-design, quality-gating, and governance problem as it is a model or hardware problem.

• Modest Claims: The authors explicitly state that the results are "serving-stack and workload-shape evidence on public synthetic data," not a disclosure of proprietary financial-institution performance. The quality observations are described as "small pilot slices intended for operational triage," not claims of state-of-the-art task quality.
• Operational Impact: The study demonstrates that standard LLM serving configurations leave major performance gains on the table. By aligning the serving stack with the specific "shape" of compliance workloads (prefix reuse, short structured outputs, multi-adapter needs), institutions can achieve high-throughput, low-latency inference while maintaining strict data governance and auditability.
• Future Directions: The paper identifies prefill/decode disaggregation and automated configuration search as natural next frontiers for scaling regulated workloads, noting that current advanced extensions (like cross-worker LMCache) are promising but not yet universal production defaults.

In conclusion, the paper provides a blueprint for building a compliance-grade LLM serving stack that prioritizes workload-specific optimizations and rigorous quality gates over raw model selection, enabling the safe and efficient deployment of LLMs in high-stakes financial domains.