🤖 AI

Beyond Next-Token Prediction: An RLVR Proof of Concept for Tool-Use Agents on Atlassian Workflows

This paper presents a proof of concept demonstrating that Reinforcement Learning with Verifiable Rewards (RLVR) significantly improves small language models' ability to execute complex, multi-step tool calls in synthetic Atlassian (Jira and Confluence) environments, achieving near-perfect success rates on most scenarios without relying on live APIs or human feedback.

Original authors: Karthikeya Aditya Vissa, Sankalp Mane, Ananya Mantravadi, Harshit Rajgarhia, Abhishek Mukherji

Published 2026-07-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Karthikeya Aditya Vissa, Sankalp Mane, Ananya Mantravadi, Harshit Rajgarhia, Abhishek Mukherji

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, well-read librarian (the AI) who is great at predicting the next word in a sentence. If you ask, "The sky is...", they will confidently say "blue." But if you ask them to actually do something in a complex office system—like "Create a new project ticket and link it to a specific page"—they often stumble. They might know the words to say, but they get the details wrong, like putting the wrong ID number in the wrong box or forgetting to check if the box exists first.

This paper is about teaching that librarian how to actually do the job, not just talk about it, using a specific training method called RLVR (Reinforcement Learning with Verifiable Rewards).

Here is the breakdown of their experiment in everyday terms:

1. The Problem: The "Autopilot" vs. The "Pilot"

Large Language Models (LLMs) are trained to be like autopilots that predict the next word. They are excellent at writing stories or emails. But in a corporate office (specifically using Atlassian tools like Jira and Confluence), success isn't about writing a pretty sentence; it's about hitting the exact right buttons in the exact right order with the exact right data.

The Old Way: You ask the AI to "Create a sub-task." The AI might say, "Okay, I'll create a sub-task," but it forgets to assign it to a person or uses the wrong project code. It looks fluent, but the task fails.
The Goal: The authors wanted to see if they could train the AI to stop just "guessing the next word" and start "acting like a pilot" who checks the instruments before flying.

2. The Solution: A "Video Game" Training Ground

Instead of letting the AI practice on the real, live company servers (which is risky and slow), the authors built a perfect digital twin (a video game version) of the Jira and Confluence systems.

The Simulator: This "game" looks and acts exactly like the real software, but it's safe. If the AI makes a mistake, nothing breaks.
The Referee (The Reward System): This is the secret sauce. Usually, you need a human to grade the AI's work. Here, they built a strict, automated referee.
- If the AI puts the right data in the right box? Points.
- If the AI checks if a page exists before trying to edit it? Bonus points.
- If the AI tries to use a tool that doesn't exist or forgets a required step? Penalty points.
- Crucially: The referee doesn't need a human to look at the screen. It just checks the math and the code.

3. The Training: Trial and Error

They took two versions of an AI (a smaller 1.7B model and a larger 4B model) and let them play in this simulator thousands of times.

The AI tries to complete a task (like "Create a page").
It fails, gets a low score from the referee, and tries again.
It slowly learns: "Oh, I got points when I checked the parent page first. I lost points when I forgot the project code."
This is Reinforcement Learning: The AI learns by getting rewards for good behavior and penalties for bad behavior.

4. The Results: From "Okay" to "Perfect"

They tested the AI before and after this training on five different office tasks.

Before Training: The AI was decent at simple tasks but terrible at complex ones. For example, on creating a new Confluence page, the untrained AI only got about 35% of the points. It kept making small, costly mistakes.
After Training: The trained AI became nearly perfect. On that same page-creation task, it jumped to 100%.
The Big Win: The training was most effective on the hardest tasks (the ones with the most rules and data fields). The AI learned to follow the strict "check-then-do" pattern required by these systems.

5. The Catch (Limitations)

The authors are very honest about what this doesn't do yet:

It's not magic for everything: They had to hand-craft the "referee rules" for these specific tasks. You can't just plug this into any software in the world without building a new referee for it first. It's like having a coach who is amazing at teaching soccer but hasn't learned basketball yet.
One task was too easy: For one specific task (changing a ticket status), the AI was already perfect before training, so the training didn't add any value there.

The Bottom Line

This paper proves that you can take a standard AI, put it in a safe, simulated office environment, and use a strict, automated referee to teach it how to follow complex rules. It transforms the AI from a "chatty bot" that sounds good but fails at the details, into a "reliable worker" that gets the job done perfectly. However, this requires building a custom "training gym" for every specific type of software you want it to use.

Technical Summary: Beyond Next-Token Prediction: An RLVR Proof of Concept for Tool-Use Agents on Atlassian Workflows

Problem Statement
Large Language Models (LLMs) are fundamentally optimized for next-token prediction on internet-scale text, a objective that misaligns with the requirements of niche enterprise SaaS workflows. In these environments, success depends on executing specific API calls with precise nested arguments in a strict order. The paper argues that this objective mismatch manifests as "silent failures" in tool-use agents: missing required fields, hallucinating non-existent tools, or terminating prematurely after a single read operation. While existing literature addresses tool invocation via supervised fine-tuning or Reinforcement Learning (RL) on open-web or broad agentic benchmarks, there is a lack of focus on narrow, schema-heavy enterprise workflows dominated by "check-then-mutate" patterns.

Methodology
The authors propose a proof of concept utilizing Reinforcement Learning with Verifiable Rewards (RLVR) applied directly within synthetic, schema-faithful copies of target environments. The approach eliminates the need for live APIs, learned reward judges, or human-in-the-loop labeling.

Synthetic Environments: The team constructed five synthetic environments emulating the Jira REST v3 and Confluence v2 APIs. These environments maintain schema fidelity (matching public contracts), are stateful but resettable between rollouts to prevent pollution, and are fully deterministic.
Verifiable Reward Functions: Instead of relying on a learned reward model, correctness is determined by programmatic checkers that inspect the tool-call trace. The reward function ( $R \in [0, 1]$ $R \in [0, 1]$ ) is decomposed into three additive components:
1. Per-argument correctness (R1): Small constants awarded for matching expected argument values (e.g., specific project keys, parent IDs).
2. Structural bonuses (R2): Rewards for adhering to the validate–mutate–verify pattern (e.g., performing a get_* call before a create_* call).
3. Penalties (R3): Subtractions for destructive behaviors, such as missing required calls, invalid payload shapes, duplicate calls, hallucinated tool names, or exceeding call budgets.
Training Loop: The authors employ Group Relative Policy Optimization (GRPO) on Qwen3-1.7B and Qwen3.5-4B models. The training loop involves generating multiple rollouts per prompt, scoring them against the verifiable reward functions, and updating the policy parameters. Training runs on modest hardware (single or dual Nvidia RTX PRO 6000 GPUs) and utilizes early stopping based on reward convergence.

Key Contributions

Synthetic Atlassian Suite: A set of five schema-faithful environments covering Jira ticket transitions, sub-task creation, Confluence page creation, page labeling, and a cross-product scenario.
Reward Design: A framework for decomposing complex tool-use correctness into verifiable, programmatic signals (argument matching, structural ordering, and penalty enforcement) computable entirely from tool-call traces.
Baseline and Training Results: A comprehensive evaluation comparing prompted baselines (Qwen3-1.7B and Qwen3.5-4B) against RLVR-trained policies using identical reward checkers.
Training Recipe: An end-to-end GRPO training recipe that achieves convergence in tens to low hundreds of generation batches on small models.

Results
The study evaluates performance on five scenarios, noting that the "ticket-transition" scenario is a transparency control where the reward function saturates (the prompted 4B model already achieves a perfect score).

Performance Gains: On the four non-degenerate scenarios, the RL-trained policy significantly outperforms the prompted baselines.
- Confluence Page Creation: The most dramatic improvement, rising from a baseline of 0.35 to 1.00 (+0.65).
- Confluence Page Labeling: Improved from 0.52 to 0.95 (+0.43).
- Jira Sub-task Creation: Improved from 0.68 to 1.00 (+0.32).
- Cross-Product (Jira + Confluence): Improved from 0.92 to 1.00, closing the residual gap to achieve perfect rollout success on all 12 evaluation prompts.
Training Dynamics: The models reached an average reward of $\ge 0.95$ within 100 generation batches for the from-scratch scenarios. The reward variance (min/max across the generation group) contracted over training as the policy stopped emitting low-reward rollouts.

Significance and Limitations
The paper positions this work as a preliminary proof of concept demonstrating that outcome-optimized small models can be trained for niche enterprise APIs using cheap, synthetic substrates and verifiable rewards.

The authors explicitly foreground two limitations to temper expectations:

Scalability of Reward Design: The hand-crafting of verifiable rewards does not currently scale to the full surface of the Atlassian public API or broader enterprise ecosystems; it is effective only for the handful of endpoints reported.
Reward Saturation: One scenario (ticket-transition) possesses a reward shape that is too simple, allowing the prompted 4B model to max out immediately, rendering RL unnecessary for that specific task.

The authors conclude that while the recipe works on this specific substrate, it does not yet establish a generalizable solution for all enterprise tool-use tasks without significant manual reward engineering.