💻 computer science

Hybrid Siamese Deep Learning Framework for Duplicate Question Detection using Contextual and Lexical Feature Fusion

This paper proposes a hybrid Siamese deep learning framework that fuses contextual BERT embeddings, lexical Glove-based CNN-BiLSTM features, and handcrafted linguistic metrics to effectively detect duplicate questions on Community Question Answering platforms, achieving 85.03% accuracy on the Quora Question Pairs dataset.

Original authors: Meherzadi Muntaha, Muhammad Junaid Asif, Naveed Hussain, Rana Fayyaz Ahmad

Published 2026-07-29

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Meherzadi Muntaha, Muhammad Junaid Asif, Naveed Hussain, Rana Fayyaz Ahmad

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are walking through a massive, bustling library where millions of people are shouting questions into the air every second. Some ask, "How do I bake a cake?" while others scream, "What's the best way to make a sponge cake?" Even though the words are different, they are asking the exact same thing. In the world of computers, this is called "Duplicate Question Detection." It's a branch of Artificial Intelligence (AI) and Natural Language Processing (NLP) that tries to teach machines to understand that two sentences can look totally different but mean the same thing.

To do this, computers usually use two main tricks. First, they look at the "dictionary" meaning of words, like checking if two books share the same vocabulary. Second, they try to understand the "context" or the vibe of the sentence, realizing that "I'm feeling blue" doesn't mean the sky is raining paint, but that someone is sad. The challenge is that sometimes computers get too focused on the dictionary and miss the vibe, or they get too lost in the vibe and miss the specific words. This paper tackles that problem by building a super-smart system that uses both tricks at the same time to spot duplicate questions on sites like Quora, saving users from reading the same answer twice and helping moderators keep the library tidy.

The Two-Headed Detective

The researchers behind this study, led by Meherzadi Muntaha and her team, built a "hybrid" detective system to solve the duplicate question mystery. Think of it as a team of two detectives working side-by-side, each with a different superpower, who then combine their notes to solve the case.

Detective A: The Context Master (The BERT Stream)
The first detective, named "Model A," is an expert at understanding the deep meaning and context of a sentence. It uses a powerful tool called BERT, which is like a super-reading machine that has read almost the entire internet. This detective doesn't just look at the words; it understands how they fit together. If someone asks, "How do I fix a flat tire?" and another asks, "What's the best way to repair a deflated wheel?", Detective A realizes that "fix" and "repair" and "flat" and "deflated" are dancing to the same tune, even if the words are different. It uses a special neural network structure called a Siamese Neural Network, which is like a mirror that looks at both questions at the same time to see if they are reflections of each other.

Detective B: The Word Counter (The GloVe Stream)
The second detective, "Model B," is a bit more old-school but incredibly sharp at spotting patterns in the words themselves. It uses GloVe embeddings, which are like a giant map of how words usually hang out together. This detective looks for specific word overlaps and the order of words. It also uses a 1D-CNN (a pattern-spotter) and a BiLSTM (a memory keeper) to remember the sequence of words. It's great at catching things like, "How do I bake a cake?" and "How do I bake a cake?" where the words are almost identical.

The Secret Sauce: Handcrafted Clues
But the team didn't stop there. They realized that sometimes the computers miss the obvious. So, they added a third layer of "handcrafted" clues. These are simple, human-made math tricks that count things like:

How many words do the two questions share?
What is the longest string of letters they have in common?
How similar do the sentences look if you squint at them (fuzzy matching)?

The Big Mix-Up
Here is where the magic happens. The team took the deep, smart understanding from Detective A, the pattern-spotting skills from Detective B, and the simple math clues from the handcrafted layer, and smashed them all together into one giant "feature fusion." Imagine taking a high-definition photo, a sketch, and a written description of a suspect and feeding them all into a single brain. This combined brain then makes the final decision: Are these two questions duplicates or not?

What They Found

The team tested their new "Hybrid Siamese Deep Learning Framework" on the Quora Question Pairs (QQP) dataset, which is a massive collection of over 400,000 pairs of questions labeled by humans as either duplicates or not. They split this data so 80% was used to teach the model and 20% was used to test it.

The results were quite impressive. The hybrid model managed to get an accuracy of 85.03%. This means it correctly identified whether questions were duplicates or not more than 85 times out of 100.

For questions that were not duplicates, it was very sure, with a precision of 91.23%.
For questions that were duplicates, it found 86.14% of them (recall) and had an F1-score of 0.8095.

The F1-score is a special number that balances how many correct guesses the model made against how many mistakes it made. The team's model scored 0.81, which is better than many of the older, single-method models they compared it against.

Why the Mix Matters

To prove that their "two-detective" approach was actually necessary, the researchers ran a little experiment called an ablation study. This is like taking apart a car to see which parts actually make it drive.

When they used only the Context Master (Model A), the accuracy dropped to 83.9%.
When they used only the Word Counter (Model B), the accuracy dropped even lower to 80.6%.
When they combined the two detectives but removed the handcrafted clues, the accuracy was 84.4%.
But when they used all three (Context + Word Patterns + Handcrafted Clues), the accuracy jumped to the final 85.03%.

This suggests that while the deep learning models are powerful, they still benefit from the simple, explicit math of the handcrafted features. The combination allows the system to be robust, handling both the subtle changes in meaning and the obvious word overlaps.

The Bottom Line

The paper concludes that this hybrid approach is a strong, scalable way to handle the messy, real-world problem of duplicate questions. It doesn't claim to have solved the problem perfectly—there were still some mistakes, like confusing questions that look similar but mean different things (false positives) or missing questions that are very cleverly rephrased (false negatives). However, by fusing contextual understanding, lexical patterns, and simple linguistic rules, the team showed that a "hybrid" system is more effective than relying on just one type of AI brain. It's a reminder that sometimes, the best way to understand human language is to use every tool in the toolbox at once.

Problem Statement

Community Question Answering (CQA) platforms like Quora, Reddit, and Stack Overflow face a significant challenge with the proliferation of duplicate questions. While these questions may differ in wording, structure, or lexical form, they often share the same semantic intent. This redundancy leads to poor information retrieval, increased moderation labor, and a degraded user experience.

Current manual moderation systems on platforms like Quora (e.g., "Merge Questions") are slow, resource-intensive, and often rely on user reputation rather than automated semantic analysis. Existing automated solutions often struggle to balance deep contextual understanding with explicit lexical similarity, particularly when dealing with paraphrased questions that lack direct word overlap. The paper posits that single-stream deep learning models or traditional machine learning approaches relying solely on handcrafted features are insufficient for robust duplicate detection in large-scale, dynamic environments.

Methodology

The authors propose a Hybrid Siamese Neural Network (SNN) framework designed to detect duplicate questions by fusing three distinct types of features: contextual embeddings, static lexical embeddings, and handcrafted linguistic features. The architecture consists of two parallel sub-models (Stream A and Stream B) that process question pairs independently before fusing their outputs.

1. Data Preprocessing:
The study utilizes the Quora Question Pairs (QQP) dataset, containing over 400,000 labeled question pairs. Preprocessing steps include:

Text normalization (lowercasing, removing punctuation/special characters).
Converting symbolic representations (e.g., "$") to text.
Handling missing values and tokenization.

2. Model A: Contextual Semantic Encoder (BERT-based)

Embedding: Uses BERT-base (WordPiece tokenizer) to generate deep contextual embeddings. It incorporates token, positional, and segment embeddings.
Architecture: The BERT output (specifically the [CLS] token representation) is passed through:
- 1D-CNN: To detect salient local contextual patterns and n-grams.
- Max Pooling: To reduce dimensionality while preserving prominent activations.
- Bidirectional LSTM (BiLSTM): To capture sequential dependencies and long-range relationships in both forward and backward directions.
Goal: To learn deep semantic relationships, paraphrasing patterns, and fine-grained contextual nuances.

3. Model B: Static Lexical Encoder (GloVe-based)

Embedding: Uses pre-trained GloVe embeddings (300-dimensional) via a rule-based tokenizer that preserves complete words.
Architecture: The embedding sequence is processed through:
- 1D-CNN: To identify local lexical patterns (bi-grams, trigrams) and fixed expressions.
- Max Pooling: To retain significant features.
- Bidirectional LSTM (BiLSTM): To model sequential dependencies and grammatical structures.
Goal: To capture global lexical semantics and sequential information based on static co-occurrence statistics.

4. Handcrafted Feature Extraction
To complement neural embeddings, the model explicitly calculates linguistic similarity metrics:

Word Overlap: Count of shared unique words and word-share ratio.
Longest Common Subsequence (LCS): To identify sequential overlapping patterns.
Fuzzy Similarity: Scores such as Fuzzy Q-ratio and partial-ratio to detect soft lexical matches.

5. Feature Fusion and Classification

The output vectors from Model A ( $z_A$ ), Model B ( $z_B$ ), and the handcrafted feature vector ( $f$ ) are concatenated into a single representation ( $z$ ).
An Attention Mechanism is applied to highlight discriminative semantic dimensions and suppress noise.
The refined vector passes through a dense feed-forward network with Dropout regularization to prevent overfitting.
A Softmax output layer performs binary classification (Duplicate vs. Non-Duplicate).

Training Configuration:

Optimizer: AdamW.
Hardware: NVIDIA GeForce RTX 4070 Ti.
Split: 80:20 stratified training/validation split.
Loss Function: Cross-entropy.

Key Contributions

The paper outlines six primary contributions:

Novel Hybrid Architecture: A dual-stream SNN framework that outperforms single-model approaches by integrating heterogeneous feature sources.
Context-Aware Sub-model (SNNA): A BERT-based stream designed to overcome the limitations of conventional lexical similarity by capturing deep semantic relations and paraphrasing.
Static Lexical Sub-model (SNNB): A GloVe/CNN/BiLSTM stream that maintains global lexical semantics and sequential stability.
Explicit Linguistic Features: The integration of handcrafted features (LCS, fuzzy matching, word overlap) to represent structural similarities that neural embeddings might miss.
Feature Fusion Mechanism: A developed method to effectively concatenate contextual, static, and handcrafted features into a unified representation space, enhancing discriminative ability.
Empirical Validation: Comprehensive experiments on the QQP dataset demonstrating superior performance over classical machine learning baselines and single-stream deep learning models.

Results

The model was evaluated on the QQP dataset using Accuracy, Precision, Recall, and F1-score.

Overall Performance: The proposed hybrid model achieved an Accuracy of 85.03% and an F1-score of 0.81.
Class-Specific Performance:
- Non-Duplicate (Class 0): Precision of 91.23%, Recall of 84.38%, F1-score of 87.67%.
- Duplicate (Class 1): Precision of 76.36%, Recall of 86.14%, F1-score of 80.95%.
Confusion Matrix: The model correctly identified 32,283 true negatives and 19,292 true positives.
Ablation Study: The study compared the full model against variants (Model A only, Model B only, and A+B without handcrafted features). The full hybrid model outperformed all variants, confirming that the combination of contextual embeddings, static embeddings, and handcrafted features provides a positive, cumulative influence on performance.

Significance and Claims

The authors claim that this research provides a scalable and robust solution for duplicate question detection in large-scale CQA systems. The significance of the work lies in its demonstration that hybrid representation learning—combining the deep semantic understanding of transformers (BERT), the stable lexical grounding of static embeddings (GloVe), and explicit structural cues (handcrafted features)—is superior to relying on a single representation paradigm.

The paper asserts that this approach effectively addresses the "semantic gap" where questions are paraphrased but lexically distinct, thereby reducing redundancy, improving user experience, and optimizing moderation resources. The study concludes that such hybrid deep learning frameworks are essential for advancing intelligent text similarity modeling in real-world NLP applications.