💻 computer science

SentTrack: Sentiment-Driven Bottleneck Detection in GitHub Issue Repositories

This paper introduces SentTrack, a dual-lens framework that leverages large language models and sentiment analysis to detect socio-technical bottlenecks in GitHub issue repositories by identifying stagnation and resolution gaps earlier than traditional methods.

Original authors: Xinyu Hu, Ali Behbahani, Daniel Moon, Yaren Dogan, Nasir U. Eisty

Published 2026-06-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xinyu Hu, Ali Behbahani, Daniel Moon, Yaren Dogan, Nasir U. Eisty

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a massive, bustling construction site where hundreds of workers are trying to build a complex digital skyscraper (the software). In the real world, workers shout across the site, leave notes on clipboards, and argue over blueprints. In the digital world, this happens in GitHub Issues—long, messy text threads where developers report bugs, ask questions, and debate how to fix things.

The problem is that the "foremen" (project maintainers) are often too busy looking at the bricks and mortar (code metrics) to notice that the workers are actually shouting at each other, getting stuck in arguments, or spinning their wheels. They only realize something is wrong when the building stops moving entirely.

SentTrack is a new tool designed to act like a smart, empathetic foreman who listens to the workers' conversations to spot trouble before the work grinds to a halt.

Here is how it works, broken down into simple parts:

1. The Two "Lenses" (The Dual-Lens Framework)

SentTrack looks at the construction site through two different pairs of glasses at the same time:

The Horizontal Lens (The "What"): This lens looks at the topics being discussed.
- The Problem: GitHub threads are messy. They are full of "machine noise"—long lists of error codes, computer logs, and technical jargon that look like gibberish to a human reader. It's like trying to hear a conversation in a room where someone is constantly screaming the alphabet.
- The Fix: SentTrack uses a super-smart AI (a Large Language Model) to act as a translator. It strips away the screaming alphabet (the error logs) and rewrites the conversation into a clean, plain-English summary. It then groups similar complaints together. For example, it realizes that "the screen flickers," "the display glitches," and "the window won't load" are all about the same problem: Visual Rendering.
- The Result: Instead of seeing 16,000 different complaints, it groups them into about 600 clear categories of problems.
The Vertical Lens (The "How"): This lens looks at the flow of the conversation over time.
- The Problem: Just because people are talking doesn't mean they are solving anything. A thread could go on for weeks with people arguing, asking questions that get no answer, or just saying "I don't know."
- The Fix: SentTrack reads every comment in a thread and labels it like a traffic light:
  - Green (Agree/Build): "Yes, that's a good idea," or "Here is the fix."
  - Yellow (Chat): "Thanks," or "Hello."
  - Red (Disagree/Elicit): "This is broken," "I disagree," or "Can you explain this?"
- The Result: It tracks the journey. If a conversation starts with a question (Red) and ends with a solution (Green), the road is clear. But if it starts with a question and ends with more questions or arguments, the road is stuck.

2. The "Bottleneck Score" (The Dashboard)

Once SentTrack has listened to the conversations and mapped the topics, it calculates a Bottleneck Score for every issue. Think of this as a "Stress Meter" for the project.

It doesn't just look at one thing. It combines four factors:

Negativity: Are people angry?
Stagnation: Is the conversation going in circles?
The Resolution Gap: Did the thread end without a solution?
Length: Is the thread unnecessarily long?

The Big Discovery:
The paper analyzed about 9,000 threads from a popular software project called AvaloniaUI. They found something surprising:

49% of the conversations ended in a "stuck" state (arguments or unanswered questions).
Only 13% actually reached a resolution.
The most dangerous signal wasn't anger. The biggest bottleneck wasn't that people were fighting; it was that they were talking for a long time without ever finishing the job. The most common "dangerous" thread was one that was Long, Stuck, and Unresolved.

3. What the Tool Actually Does

SentTrack gives the project managers a prioritized list. Instead of guessing which bug to fix first, they can look at the dashboard and see: "Hey, this thread has been going on for three weeks, nobody has answered the questions, and the mood is getting tense. Let's jump in and fix this before the whole team stops working."

Summary

In short, SentTrack is a tool that listens to the "chatter" of software developers. It cleans up the technical noise, groups similar complaints, and watches how the conversations move. It tells project leaders: "Don't just look at the code; look at the conversation. Here is where the team is stuck, here is where they are angry, and here is where the work is slowing down."

It's like having a tool that can hear the difference between a productive brainstorming session and a meeting where everyone is just talking in circles, helping the team get back to building the skyscraper.

Technical Summary: SentTrack – Sentiment-Driven Bottleneck Detection in GitHub Issue Repositories

Problem Statement

Software engineering teams increasingly rely on GitHub issue threads to coordinate work, report bugs, and negotiate technical decisions. However, existing repository health tools predominantly focus on code metrics, treating issue discussions as secondary. Current methods for identifying workflow friction—such as periodic retrospectives or manual label curation—are reactive and slow to detect emerging problems. Furthermore, automated analysis of GitHub issues faces two specific challenges:

Inconsistent Metadata: Repository labels (e.g., bug, enhancement) are often sparse, inconsistent, or lack context regarding why issues remain unresolved.
Mixed-Media Noise: Issue threads contain significant "machine-generated noise" (stack traces, log dumps, code snippets) that dominates text volume but carries no user intent, causing standard language models to extract topics rooted in code artifacts rather than developer concerns.

This paper addresses the gap in identifying socio-technical bottlenecks by asking three research questions (RQs):

RQ1: How can an automated framework identify workflow inefficiencies using real-time conversational data?
RQ2: Can sentiment analysis surface risk signals earlier than traditional label-based management?
RQ3: How can human narrative be effectively isolated from machine-generated noise to ensure topic extraction reflects actual developer intent?

Methodology: The SentTrack Framework

SentTrack is a dual-lens framework applied to the AvaloniaUI open-source repository (approximately 9,000 issue threads). It combines a horizontal pipeline for global topic modeling and a vertical pipeline for per-thread interaction analysis.

1. Horizontal Pipeline (Topic Modeling)

This pipeline processes the issue title and body to identify macro-level trends and recurring technical themes.

Intent-Focused Summarization: A Large Language Model (LLM) translates raw issue text into clean summaries. The LLM is instructed to retain developer intent while suppressing machine artifacts (file paths, hex codes, stack traces).
Context-Aware Topic Extraction: From the clean summaries, the pipeline extracts mid-level concern phrases (3–6 word noun phrases). These are guided by a static project profile to ensure domain relevance (e.g., "Rendering," "Cross-platform input") and sit at an abstraction level between generic labels and specific error tokens.
Semantic Clustering: Extracted phrases are embedded using an instruction-tuned encoder, reduced via UMAP, and clustered using HDBSCAN. This approach identifies density-based clusters without requiring a predefined cluster count.

2. Vertical Pipeline (Thread Dynamics)

This pipeline treats each issue as a chronological sequence of comments rather than a single document.

ABCDE Classification: Each comment is classified into one of five interaction categories based on the collaborative interaction framework:
- A (Agree): Confirmation or convergence.
- B (Build): Added evidence, proposed fix, or explanation.
- C (Chat): Neutral or weakly task-moving text.
- D (Different Perspective): Disagreement, concern, or blocker.
- E (Elicit): Question or clarification request.
Outcome Inference: The thread outcome is determined by the final interaction in the sequence. Sequences ending in A or B are classified as resolved; D or E as stagnant; and C as neutral.

3. Bottleneck Detection Engine

A weighted scoring engine fuses data from both pipelines to generate an interpretable bottleneck score ( $B_i$ ) for each issue:
$B_i = 0.35N_i + 0.30S_i + 0.25R_i + 0.10L_i$
Where:

$N_i$ : Aggregated negativity (average, max, and ratio of negative comments).
$S_i$ : Binary stagnation indicator.
$R_i$ : Resolution gap score.
$L_i$ : Normalized thread length.

The engine also assigns compound bottleneck categories (e.g., Negative, Stagnant, Long, Unresolved) to help maintainers prioritize triage.

Key Results

Applied to the AvaloniaUI corpus, SentTrack produced the following findings:

Topic Clustering: From the first 3,608 issues, the pipeline generated 613 semantic clusters. However, a discovery-rate analysis revealed a structural limitation: the LLM generated roughly two unique topic variants per issue (e.g., "XAML previewer startup failure" vs. "designer process crash"), resulting in over 1,800 clusters for the full corpus. This suggests the pipeline produces an order of magnitude more clusters than are actionable for maintainers, indicating a need for vocabulary constraints or a topic-reduction step.
Thread Outcomes: Across the full corpus, 49% of threads ended in stagnation (D or E), 38% in neutral commentary, and only 13% reached resolution (A or B).
Bottleneck Signals: The resolution gap was identified as the dominant bottleneck signal. While negativity is a factor, the most frequent compound bottleneck category was "Stagnant and Long and Unresolved." This indicates that the primary risk is not emotional negativity, but rather prolonged, directionless discussions that fail to converge.
Diagnostic Metrics: The Question-Resolution Rate (QRR) and Concern-Capture Rate (CCR) were lower than desired, indicating that questions and concerns often appear in threads but are not followed by constructive resolution.

Significance and Claims

The paper claims that SentTrack successfully demonstrates that real-time conversational data can surface workflow friction that traditional labels and periodic reviews miss.

Addressing RQ3: The framework successfully isolates human narrative from machine noise through intent-focused LLM summarization, allowing downstream topic extraction to reflect developer concerns rather than code artifacts.
Addressing RQ1 & RQ2: The vertical pipeline proves that simple, rule-based ABCDE classification can produce interpretable thread-level signals without heavy LLM inference, revealing that conversation structure is as critical as text content for reliability.
Unified Signal: The primary contribution is an interpretable, end-to-end pipeline that links technical topics, thread outcomes, and bottleneck risk scores. This provides maintainers with a unified view of what is breaking down and where in the community it is happening.

The authors conclude that while the current implementation suffers from topic over-fragmentation, the core premise holds: combining horizontal topic modeling with vertical thread analysis yields earlier and more interpretable signals of workflow breakdown than code metrics or manual labels alone. The immediate path forward involves consolidating the cluster space through a dedicated topic-reduction phase.