💬 NLP

Extracting Problem and Method Sentence from Scientific Papers: A Context-enhanced Transformer Using Formulaic Expression Desensitization

This paper proposes a context-enhanced transformer model utilizing formulaic expression desensitization for data augmentation to effectively extract problem and method sentences from scientific papers, achieving superior performance over baselines while demonstrating the unsuitability of large language model-based in-context learning for this task.

Original authors: Yingyi Zhang, Chengzhi Zhang

Published 2026-06-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yingyi Zhang, Chengzhi Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the world of scientific research as a massive, ever-growing library containing billions of books. Each book is a scientific paper, and inside them, researchers describe the problems they are trying to solve and the methods they used to solve them.

The challenge is that humans can't read all these billions of books fast enough. We need a robot (an AI) to read them for us and pull out just the sentences that say, "Here is the problem," and "Here is how we fixed it."

However, this paper argues that previous robots were a bit too "lazy" and "stereotypical." They relied too much on specific phrases (like "we used a method" or "the problem is") to guess what a sentence meant. If a sentence had those magic words, the robot guessed "Method!" even if the sentence was actually about something else. This made the robot bad at understanding new books it hadn't seen before.

Here is how the authors fixed this, using three main tricks:

1. The "Desensitization" Diet (Formulaic Expression Desensitization)

The Problem: The robot was like a student who memorized that every time they saw the word "for," it meant a solution was coming. So, when the robot saw "a parser for German," it immediately shouted "Method!" even though the sentence was just describing a tool, not a method being used.

The Solution: The authors created a special "diet" for the robot's training data. They took the training sentences and swapped out those overused, magic phrases (like "for," "used," "present") with blanks or random words.

The Analogy: Imagine teaching a child to recognize a dog. If you only show them pictures of Golden Retrievers, they might think all dogs are Golden Retrievers. If you show them a Golden Retriever but cover the fur with a blanket, they have to learn to recognize the dog by its shape and eyes, not just its fur.
The Result: By "desensitizing" the robot to these specific phrases, it learned to actually understand the meaning of the sentence rather than just spotting a keyword. This made the robot much smarter at generalizing to new papers.

2. The "Context Detective" (Context-Enhanced Transformer)

The Problem: Sometimes a sentence is confusing on its own.

Target Sentence: "Glosser is designed to support reading..."
Confusion: Is "Glosser" a problem or a method? On its own, it's hard to tell.
The Clue: The sentence after it says, "There are four language pairs supported by Glosser..." This tells us Glosser is a tool (a method).

The Solution: The authors built a robot that doesn't just read one sentence in isolation. It looks at the sentence before and after (the context) like a detective looking for clues.

The Analogy: Imagine you are reading a mystery novel. If you read a single line, "He picked up the gun," you might think he's a criminal. But if you read the line before, "The detective needed to prove his theory," you know he's a hero.
The Innovation: Previous robots tried to read the whole chapter at once (which was too heavy and slow) or just glued the sentences together (which created noise). This new robot uses a "spotlight" (an attention mechanism) to look at the surrounding sentences and ask, "Which parts of the neighbor sentences help me understand this specific sentence?" It filters out the noise and focuses only on the helpful clues.

3. The "Small Dataset" Struggle

The Problem: To teach these robots, humans have to manually label thousands of sentences. This is slow, expensive, and results in very small training sets. Small sets make robots "overfit"—they memorize the training data instead of learning the rules.

The Fix: The authors used the "Desensitization" trick (swapping words) to create synthetic data. They took their small dataset and generated thousands of new, slightly different versions of it. This gave the robot more to learn from without needing humans to label every single new sentence.

What Didn't Work? (The LLM Experiment)

The authors also tried using the latest, super-smart AI models (Large Language Models or LLMs) to do this job without any training, just by giving them a few examples (a technique called "In-Context Learning").

The Result: It failed miserably. The super-smart AI got confused and performed much worse than the specialized robot they built.
The Takeaway: Just because a model is "smart" and good at chatting doesn't mean it's good at this specific, technical job of finding problem and method sentences in scientific papers.

The Final Score

When they tested their new robot (the "Context-Enhanced Transformer" trained on "Desensitized" data) against the old robots:

It was significantly better at finding the right sentences.
It improved accuracy by about 3.7% on abstracts and 2.7% on full papers.
It did this without taking much longer to run.

In short: The authors taught a robot to stop relying on lazy shortcuts (specific phrases) and start paying attention to the surrounding story (context), allowing it to read scientific papers much more accurately than before.

Technical Summary: Extracting Problem and Method Sentences from Scientific Papers

1. Problem Statement

The exponential growth of scientific literature (reaching billions of papers) necessitates automated methods to identify essential components, specifically problem sentences (describing technical gaps or research subjects) and method sentences (describing processes, experimental setups, or tools). However, current approaches face significant challenges:

Data Scarcity: Manual annotation is labor-intensive, resulting in small-scale datasets (e.g., hundreds of abstracts) that limit the information available for training deep neural networks.
Overfitting to Formulaic Expressions (FEs): Due to limited data, models tend to memorize specific high-frequency patterns or "formulaic expressions" (e.g., "was used for," "this paper presents") rather than learning semantic meaning. This reliance on FEs reduces generalization capabilities, causing models to misclassify sentences that contain these phrases but lack the actual problem or method content.
Contextual Noise: While context (adjacent sentences) can enrich semantic understanding, adjacent sentences often contain irrelevant information. Naive integration of context (e.g., simple concatenation) introduces noise, degrading performance.
LLM Limitations: There is a lack of research evaluating whether Large Language Models (LLMs) using In-Context Learning (ICL) can effectively perform this extraction task without fine-tuning.

2. Methodology

The paper proposes a two-pronged approach to address data scarcity and model generalization, alongside an evaluation of LLM-based ICL.

A. FE Desensitization-Based Data Augmentation

To mitigate overfitting to specific patterns, the authors introduce Formulaic Expression (FE) Desensitization. This process involves replacing FEs in training sentences with a placeholder (e.g., "_") to force the model to rely on semantic content rather than syntactic shortcuts.

FE Selection Strategies: Three strategies are employed to identify FEs:
1. Dependency-tree-based: Uses Stanford CoreNLP to identify root nodes, parent nodes, and connecting prepositions of problem/method phrases within the dependency tree.
2. LLM-based: Uses prompts to extract FEs from sentences.
3. Open-source-based: Utilizes pre-existing FE lists from prior research (Iwatsuki & Aizawa, 2021).
Quality Discrimination: Augmented sentences are filtered using two strategies:
- True-discrimination: Keeps augmented sentences where the model correctly predicts the original label.
- Wrong-discrimination: Keeps augmented sentences where the model incorrectly predicts the label, aiming to provide "harder" examples that correct model biases.

B. Context-Enhanced Transformer

To integrate context without introducing noise, the authors propose a Context-Enhanced Transformer. Unlike sequential models that process all sentences as a sequence or concatenation models that simply join context and target sentences, this model uses a query-key-value mechanism:

Query ( $Q$ ): Derived from the concatenated vector representations of the preceding and following sentences (context).
Key ( $K$ ) and Value ( $V$ ): Derived from the target sentence.
Mechanism: The context acts as a query to attend to and weigh the importance of words within the target sentence. This allows the model to enrich the target sentence's representation with relevant context while filtering out noise (e.g., irrelevant terms in adjacent sentences).

C. LLM-Based In-Context Learning (ICL)

The study evaluates two ICL paradigms using GPT-3.5:

Sentence-level ICL: Predicts the category of individual sentences based on prompts containing definitions and demonstrations.
Paragraph-level ICL: Extracts categories from entire paragraphs.

D. Dataset Construction

Two manually annotated datasets were created:

Abstract Dataset: 305 NLP abstracts (1,440 sentences) from the SCIERC dataset.
Full-Text Dataset: 74 ACL papers (12,594 sentences).

Labeling: Three categories were defined: Problem, Method, and a new category, Co-occurrence (sentences describing both). A cross-weight-based error correction framework was used to minimize human annotation errors.

3. Key Contributions

FE Desensitization: Introduction of a novel data augmentation technique that specifically targets and desensitizes models from relying on formulaic expressions, thereby improving generalization on small datasets.
Context-Enhanced Transformer: A model architecture that effectively integrates context as a query to measure word importance in target sentences, successfully reducing noise compared to simple concatenation or sequential approaches.
New Dataset and Definitions: Manual construction of two datasets (Abstract and Full-Text) and the formal definition of "Co-occurrence sentences," addressing a gap in previous literature.
LLM Evaluation: A comprehensive evaluation demonstrating that current LLM-based ICL methods are not yet suitable for this specific extraction task compared to fine-tuned models.

4. Experimental Results

Experiments were conducted on the Abstract and Full-Text datasets using macro F1 scores as the primary metric.

Data Augmentation: The FE desensitization-based augmenter (specifically using dependency-tree-based selection) outperformed baseline augmenters (Random, TFIDF, Embedding, Back-translation, GPT).
- Improvement: It achieved a macro F1 of 73.98% on the Abstract dataset and 73.36% on the Full-Text dataset, surpassing the baseline PLM-BiLSTM-ATT by 3.36% and 1.9%, respectively.
Context-Enhanced Transformer: This model outperformed the baseline PLM-BiLSTM-ATT and the Concatenation model.
- Improvement: It achieved macro F1 scores of 73.45% (Abstract) and 72.98% (Full-Text), representing improvements of 2.83% and 1.52% over the baseline.
Combined Approach: Combining the Context-Enhanced Transformer with the FE desensitization augmenter yielded the best results.
- Improvement: Macro F1 increased by 3.71% (Abstract) and 2.67% (Full-Text) compared to the baseline.
LLM-Based ICL: Both sentence-level and paragraph-level ICL methods performed significantly worse than the fine-tuned baseline models (e.g., Sentence-level ICL achieved only ~22% macro F1 on the Abstract dataset vs. ~70% for the baseline).

5. Significance and Claims

The paper claims that its proposed methods address the critical bottleneck of small-scale datasets in scientific text mining. By reducing reliance on superficial formulaic expressions and intelligently filtering contextual noise, the proposed models achieve higher generalization and accuracy.

The authors modestly conclude that:

FE desensitization and context enhancement are effective strategies for sentence extraction tasks, particularly in low-resource scenarios.
The specific FEs extracted from the target training domain are more effective than generic open-source FEs.
Directly concatenating context with target sentences introduces noise, whereas the proposed attention-based mechanism effectively filters it.
Current LLM-based ICL is not a viable replacement for fine-tuned models for this specific task, highlighting a gap in current generative AI capabilities for structured scientific extraction.

The work suggests that these datasets and methodologies can serve as resources for future research in knowledge graph construction, academic search, and summarization, though the authors note limitations regarding the scope of experimental fields and the potential for reinforcement learning to optimize FE selection in future work.