Multimodal Structural Anomaly Detection in Invoices Using Text–Logic Fusion

Technical Summary: Multimodal Structural Anomaly Detection in Invoices Using Text–Logic Fusion

Problem Statement
Financial documents, particularly invoices, are critical to automated auditing and accounting systems. However, these semi-structured records are prone to irregularities such as missing fields, unusual formatting, and template deviations. Traditional validation methods rely heavily on rule-based arithmetic checks, which fail to detect structural anomalies like missing fields or layout inconsistencies. Conversely, text-based machine learning models often overlook the crucial numerical relationships inherent in financial documents. The core challenge addressed by this study is the detection of structural anomalies in invoices without labeled anomaly data, requiring a system that can simultaneously understand semantic text patterns and financial logical consistency.

Methodology
The authors propose an unsupervised multimodal framework that fuses textual semantics and financial logic to detect structural anomalies. The methodology proceeds through the following stages:

1. Data Preprocessing and Feature Extraction:

   • Optical Character Recognition (OCR): Text is extracted from invoice images using OCR. The raw text is cleaned (lowercasing, removing whitespace and non-informative symbols) to prepare for semantic analysis.
   • Textual Semantic Encoder: Cleaned text is converted into dense sentence-level embeddings using Sentence-BERT. These embeddings capture contextual relationships and layout information. An autoencoder architecture (encoder-decoder with ReLU activation and batch normalization) is used to learn a latent representation ($z_t$) of these semantic patterns, minimizing reconstruction loss ($L_{text}$).
   • Financial Logic Encoder: Numerical values (subtotals, taxes, totals) are extracted from the OCR output. Logic-based feature vectors are constructed to represent the deviation between expected and observed financial values (e.g., $\Delta_{line}, \Delta_{subtotal}, \Delta_{tax}, \Delta_{total}$). A separate encoder maps these logic features into a latent space ($z_l$), learning patterns of financial consistency.

2. Multimodal Fusion Architecture:

   • The latent representations from the textual encoder ($z_t$) and the logic encoder ($z_l$) are concatenated to form a unified multimodal feature vector ($z_f$).
   • A Fusion Autoencoder processes this combined vector. The network is trained to reconstruct the input fused features, learning the distribution of structurally normal invoices.
   • Anomaly Scoring: During inference, the anomaly score ($S$) is calculated as the Mean Squared Error (MSE) between the original fused feature vector and its reconstructed version. Higher reconstruction errors indicate structural deviations from the learned normal patterns.

Key Contributions

• Multimodal Fusion Approach: The study introduces a novel unsupervised framework that jointly models semantic text features and financial logic features, addressing the limitation of existing methods that treat invoices as purely textual or purely numerical.
• Unsupervised Learning: The system operates without labeled anomaly data, utilizing reconstruction error to identify irregularities such as incomplete invoices, template pages, and structurally inconsistent documents.
• Hybrid Architecture: The integration of a Textual Semantic Block and a Financial Logic Block within a single fusion autoencoder allows the model to detect both semantic layout errors and arithmetic inconsistencies simultaneously.

Experimental Results
The proposed model was evaluated on the "High Quality Invoice Images for OCR" dataset, comprising 8,181 invoice images with varying structures. The dataset was split for training (on normal samples) and testing.

• Performance Comparison: The proposed fusion model was compared against single-modality baselines (Text-Only Autoencoder, Logic-Only Autoencoder) and classical anomaly detection algorithms (Isolation Forest, One-Class SVM, Local Outlier Factor).
• Quantitative Findings: The fusion model achieved a mean anomaly score of 0.3750 with a standard deviation of 0.2962. While the Logic-Only model showed a higher mean score (0.5107), the fusion model demonstrated superior separation capabilities. The text-only and logic-only baselines were effective in their respective domains but lacked the holistic view provided by the fusion model.
• Qualitative Analysis: The anomaly score distribution indicated a clear separation between normal and anomalous samples. The fusion model exhibited higher score dispersion, suggesting a better ability to distinguish subtle inconsistencies that single-modality models might miss. The confusion matrix (based on pseudo-labels) showed a stable boundary between high-risk and low-risk samples.

Significance and Claims
The paper claims that the proposed multimodal fusion approach offers a more robust and discriminative solution for invoice anomaly detection compared to unimodal methods. By learning correlations between semantic information and financial structure, the system effectively identifies structural inaccuracies, missing information, and financial discrepancies. The authors assert that this approach is suitable for large-scale automated financial audits and document intelligence systems. The study concludes that while the current results are promising, future work could explore layout-aware embeddings and transformer-based encoders to further enhance the understanding of complex invoice structures. The research is presented as a feasible step toward reducing false positives and improving the reliability of automated financial processing.