Help! Need Advice on Identifying Advice

1. Problem Statement

The paper addresses the challenge of automatic advice identification in online text. While humans frequently seek and give advice on platforms like Reddit, advice is often mixed with non-advice content (e.g., emotional support, personal narratives, reassurance) and can be stated either explicitly or implicitly.

• The Gap: Existing Natural Language Processing (NLP) tasks, such as "suggestion mining" (e.g., SemEval-2019), focus primarily on explicit suggestions. They fail to capture implicit advice conveyed through personal narratives or discourse context.
• The Difficulty: Identifying advice requires deep understanding of semantics and discourse pragmatics. For example, a sentence like "I took my meds..." is not advice in isolation, but in response to a question about managing depression, it functions as implicit advice.

2. Methodology

A. Data Collection and Annotation

The authors constructed a new dataset from two distinct Reddit communities:

1. r/AskParents: A niche forum for parenting advice (less moderated, higher reliance on personal narrative).
2. r/needadvice: A general advice forum (more heavily moderated, broader topics).

• Scale: 684 posts containing 18,456 sentences.
• Annotation Task: Crowdworkers on Amazon Mechanical Turk highlighted the longest contiguous spans of text constituting advice within comment trees. They were provided with the full context (the original question and the reply thread) to identify both direct and indirect advice.
• Aggregation: The Dawid-Skene (DS) algorithm was used to aggregate labels from multiple annotators, weighting them by estimated competence. This was validated against "expert" annotations by linguists, showing higher agreement than simple majority voting.
• Statistics: Only ~26% to 37% of sentences in replies were labeled as advice, highlighting the sparsity of the target signal.

B. Preliminary Analysis

The authors analyzed the linguistic characteristics of the data:

• Discourse Modes: Advice is often delivered via personal narratives (especially in r/AskParents and Mental Health topics) or argumentative modes.
• Non-Advice Content: A significant portion of replies consists of sentiment expression, support, or reasoning, which complicates classification.
• Lexical Differences: Advice sentences contain more verbs and action-oriented lemmas, while non-advice sentences rely more on sentiment words (e.g., "luck," "sorry," "thanks").

C. Modeling Approach

The task was framed as a binary sentence classification problem (Advice vs. Non-Advice).

• Baselines:
  • Rule-based models using keywords (e.g., "suggest," "recommend") and Part-of-Speech (POS) patterns.
  • Top-performing models from SemEval-2019 Task 9.
• Proposed Models (BERT-based):
  The authors fine-tuned BERT (Bidirectional Encoder Representations from Transformers) with three input configurations to test the impact of context:
  1. BERTsent: Input is the sentence only.
  2. BERTsent+q: Input is the sentence + the question title and the last 3 sentences of the question body.
  3. BERTsent+c: Input is the sentence + the remainder of the reply (context from the same user).
  • Note: They also tested non-fine-tuned BERT embeddings (BERTnoft).

3. Key Results

Performance Comparison

• Rule-based vs. Neural: Fine-tuned BERT models significantly outperformed rule-based baselines and non-fine-tuned embeddings.
  • r/needadvice: BERTsent achieved an F1 of 77.8%, compared to 57.2% for the best rule-based baseline.
  • r/AskParents: BERTsent achieved an F1 of 51.9%, compared to 44.6% for the rule-based baseline.
• Domain Difficulty: Models performed substantially better on r/needadvice than r/AskParents. The authors attribute this to r/AskParents having a higher density of implicit advice via personal narratives, which are harder to detect.

Impact of Context

• Surprising Finding: Adding context (either the question or the rest of the reply) did not improve performance and, in some cases (specifically BERTsent+q), degraded it significantly.
• Reasoning: The authors hypothesize that the standard [CLS] SENT [SEP] CONTEXT architecture may struggle with discourse incoherence when the question and answer are written by different users with different styles.

Generalizability

• Models trained on the niche r/AskParents dataset transferred reasonably well to the general r/needadvice dataset, suggesting that core features of advice are learnable across domains.
• However, models struggled specifically with personal narrative sentences (containing pronouns like "I," "my," "we"), showing lower precision and high variance.

4. Key Contributions

1. New Dataset: The first large-scale, sentence-level annotated dataset for advice identification that explicitly includes implicit advice and diverse discourse strategies (narrative vs. argumentative).
2. Linguistic Insights: A detailed analysis revealing that advice is often embedded in personal narratives and that current models fail to capture these implicit cues.
3. Benchmarking: Established strong baselines using BERT and demonstrated that while pre-trained models capture surface-level syntactic cues (like modals), they struggle with discourse-level pragmatics.
4. Resource Availability: The authors released the dataset and code to facilitate future research in advice mining and NLG.

5. Significance and Future Directions

• Practical Impact: Improved advice identification can drastically increase the efficiency of advice-seeking systems and enhance Natural Language Generation (NLG) systems that aim to provide helpful, human-like advice.
• Theoretical Contribution: The work highlights the limitations of current Transformer architectures in handling discourse pragmatics and implicit intent.
• Future Work: The authors suggest that future models need to better integrate discourse-level information to distinguish between support and advice, particularly in narrative contexts. They also call for architectures that can effectively utilize question-reply context without suffering from incoherence.

In summary, this paper moves beyond simple keyword matching to tackle the complex, pragmatic nature of advice in human conversation, providing a critical resource and baseline for the NLP community.