🤖 AI

RAG4Tickets: AI-Powered Ticket Resolution via Retrieval-Augmented Generation on JIRA and GitHub Data

The paper proposes RAG4Tickets, an AI-powered framework that integrates JIRA and GitHub data using Retrieval-Augmented Generation to deliver context-aware, evidence-based ticket resolution recommendations, significantly improving accuracy and efficiency in software development workflows.

Original authors: Mohammad Baqar

Published 2026-02-06

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mohammad Baqar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a software developer working on a massive project. Every time a bug pops up, it's like a new detective case. In the past, solving these cases was like searching for a needle in a haystack made of millions of other needles. You'd have to dig through old JIRA tickets (bug reports), read thousands of developer comments, and scan through GitHub code changes to find if someone else had already solved a similar problem. Often, even if the problem was the same, the words used to describe it were different, so you'd miss the solution and waste hours reinventing the wheel.

This paper, "RAG4Tickets," proposes a smart assistant that acts like a super-powered librarian and detective combined to fix this problem. Here is how it works, broken down into simple steps:

1. The Problem: The "Lost in Translation" Library

Think of your company's history of bugs and fixes as a giant, chaotic library.

The Issue: A developer writes a ticket saying, "The app freezes when I click the button." Another developer from two years ago wrote a ticket saying, "The screen locks up during user interaction." Even though these are the same problem, a simple keyword search might miss the connection.
The Result: Developers spend too much time searching and not enough time fixing.

2. The Solution: The "Smart Librarian" (RAG)

The authors built a system called RAG4Tickets. Think of this system as a highly trained librarian who doesn't just read the titles of books but understands the meaning behind them.

Step 1: Reading and Understanding (Embeddings):
The system reads every old bug report, comment, and code change (Pull Request). It uses a special tool called Sentence-Transformers to turn these words into "meaning maps" (vectors). Instead of just looking for the word "crash," it understands that "freeze," "lock up," and "hang" all point to the same concept.
Step 2: The Super-Fast Search (FAISS):
Once the system has these meaning maps, it stores them in a massive, ultra-fast index called FAISS. Imagine this as a magical map where similar ideas are physically located right next to each other. When a new bug comes in, the system instantly finds the top 5 or 10 most similar past cases, even if the wording is totally different.
Step 3: The Detective's Report (LLM Generation):
The system doesn't just show you the old tickets; it acts like a detective summarizing the case. It takes the old solutions it found and feeds them to a Large Language Model (AI). The AI then writes a clear, step-by-step guide on how to fix the new bug, citing exactly which old code changes solved it. It's like having a senior engineer whisper, "Hey, we fixed this exact thing last year; here is the code we used."

3. How It Works in Real Life

The paper tested this system during a major upgrade of a web application (moving from React 18 to React 19).

The Scenario: Developers were struggling with a specific type of "UI freeze" caused by the new software version.
The Result: The system found old tickets and code changes related to similar freezes. It told the developers, "This looks like the issue we had before. Here is the code fix we used then."
The Outcome:
- Speed: The time it took to fix a bug dropped from 18.5 hours to 10.2 hours (a 45% improvement).
- Trust: About 68% of the time, developers accepted the AI's suggestion directly or with just a tiny edit.
- Accuracy: The system was very good at finding the right past cases (82% of the time, the right answer was in the top 5 results).

4. Why This Matters

The paper highlights that this isn't just about being faster; it's about not making things up.

No Hallucinations: Because the AI is forced to look at real, historical evidence (the old tickets and code) before it writes a solution, it is much less likely to invent fake fixes. It's like a student who must quote their textbook before answering a test question.
Knowledge Reuse: It stops companies from losing their "institutional memory." When a senior developer leaves, their knowledge isn't lost; it's locked in the system for everyone to use.

5. The Catch (Limitations)

The authors are honest about the challenges:

Old Mistakes: If the old tickets contained bad advice, the system might repeat those mistakes.
Changing Languages: If the software technology changes drastically (like a new version of React), the old "meaning maps" might become less accurate, requiring the system to be retrained.
Cost: Running this system requires powerful computers to search through millions of records instantly.

Summary

In short, RAG4Tickets is a tool that connects the dots between past problems and current solutions. It turns a chaotic pile of bug reports into a smart, searchable knowledge base that helps developers solve problems faster, with less frustration, and with the confidence that they are building on proven, real-world fixes.

Technical Summary: RAG4Tickets – AI-Powered Ticket Resolution via Retrieval-Augmented Generation

1. Problem Statement

Modern software development teams face significant delays in resolving recurring issues due to fragmented knowledge scattered across JIRA tickets, developer discussions, and GitHub pull requests (PRs). As projects scale, the exponential growth of tickets and code changes leads to information overload. Developers often struggle to locate past resolutions for semantically similar but linguistically distinct issues (e.g., matching "UI crash" with "application freeze"). Traditional supervised machine learning approaches for bug classification often fail to capture the semantic variability inherent in real-world bug reports, leading to inefficiencies in triage and resolution.

2. Methodology

The paper proposes RAG4Tickets, a Retrieval-Augmented Generation (RAG) framework designed to unify heterogeneous software artifacts into a context-aware resolution pipeline. The system operates through four primary layers:

Data Ingestion & Preprocessing: The system ingests structured data from JIRA (titles, descriptions, status, comments) and GitHub (PRs, commit messages, diffs, review comments). Preprocessing involves normalizing text, removing boilerplate, and using Abstract Syntax Tree (AST) parsers to summarize code diffs at the function level to reduce noise.
Embedding Layer: Heterogeneous artifacts are converted into dense vector representations. The framework utilizes Sentence-Transformers (specifically all-MiniLM-L6-v2 and multi-qa-MPNet-base-dot-v1) for textual data (tickets, comments). For code-specific data (PR diffs), it employs a hybrid strategy incorporating CodeBERT and GraphCodeBERT to capture structural dependencies and programming semantics. Embeddings are partitioned by artifact type to enable targeted search.
Retrieval Layer: All embeddings are indexed using FAISS (Facebook AI Similarity Search) to perform Approximate Nearest Neighbor (ANN) search. The system evaluates different index structures (Flat, IVF, HNSW) and selects HNSW for its optimal balance of speed and recall in enterprise-scale datasets (millions of vectors). Upon receiving a new ticket, the system encodes it and retrieves the top- $k$ semantically similar historical tickets, comments, and PRs.
Generation Layer: Retrieved evidence is synthesized into a structured prompt for a Large Language Model (LLM). The LLM generates grounded resolution suggestions, including step-by-step plans, hyperlinks to relevant PRs/commits, and confidence scores. This layer is designed to minimize hallucinations by strictly grounding outputs in retrieved evidence.

Deployment Integration: The framework is modular, allowing integration via JIRA plugins (for automated triage comments), GitHub Actions (for PR-linked issue surfacing), and CI/CD pipelines (for auto-triaging recurring test failures).

3. Key Contributions

The paper outlines five primary contributions:

Unified Architecture: A pipeline that ingests and links JIRA and GitHub data, establishing explicit connections between tickets and code changes.
Specialized Embedding Strategy: A tailored approach for heterogeneous artifacts, combining semantic text embeddings with code-aware embeddings (CodeBERT) to capture both natural language and structural code similarities.
Evidence-Guided Decision Pipeline: A RAG workflow where retrieved historical evidence directly guides LLM-generated resolution suggestions, ensuring traceability.
Comprehensive Evaluation: An assessment using both Information Retrieval (IR) metrics (Recall@k, Mean Reciprocal Rank) and developer productivity measures (time-to-resolution, acceptance rates).
Practical Case Study: A real-world deployment scenario involving a React 19 microservice migration, demonstrating the system's ability to handle recurring issues related to deprecated lifecycle methods and state management.

4. Experimental Results

The system was evaluated on a dataset comprising approximately 15,000 JIRA tickets and 6,500 GitHub PRs from a large-scale React 19 migration project.

Retrieval Performance: Using the HNSW index, the system achieved a Recall@5 of 82% and Recall@10 of 91%, with a Mean Reciprocal Rank (MRR) of 0.78. This indicates that relevant historical resolutions are typically found within the top five results.
Generation Quality: On a test set of 500 unseen issues, the generated solutions achieved a BLEU score of 0.47 and ROUGE-L of 0.62. Crucially, the factual consistency score was 84%, demonstrating that the LLM largely adhered to the retrieved evidence, thereby minimizing hallucinations.
Business Impact: In a deployment across three agile teams, the system yielded the following improvements:
- Resolution Time: Reduced from an average of 18.5 hours to 10.2 hours (a 45% improvement).
- Acceptance Rate: Developers directly adopted or lightly edited AI-suggested resolutions 68% of the time.
- Productivity: Developers reported a 32% self-reported productivity uplift, attributed to reduced time spent on repetitive triaging and manual searching.

5. Significance and Limitations

The paper posits that RAG4Tickets demonstrates how AI-augmented DevOps workflows can mitigate cognitive overload and enhance organizational knowledge reuse. By bridging the gap between unstructured historical data and automated resolution, the framework offers a scalable path toward context-aware engineering environments.

However, the authors maintain a modest stance regarding current limitations:

Hallucination Risk: While factual grounding is high (84%), occasional hallucinations occur when the LLM extrapolates beyond retrieved evidence.
Data Bias and Drift: The system risks propagating suboptimal patterns from legacy data and may suffer from semantic drift as frameworks evolve (e.g., React 18 to 19).
Operational Costs: Indexing large repositories and frequent LLM inference introduce compute and latency overheads.

The paper concludes that while the system shows strong potential for automating software maintenance, long-term success depends on mitigating hallucination risks through retrieval verification, ensuring model adaptability to evolving technologies, and maintaining transparency to build developer trust. Future work is proposed to include retrieval verification layers, reinforcement learning with human feedback (RLHF), and hybrid retrieval architectures combining dense and sparse search methods.