💻 computer science

ProfiLLM: Utility-Aligned Agentic User Profiling for Industrial Ride-Hailing Dispatch

ProfiLLM is an agentic LLM data pipeline that overcomes the challenges of scaling user profiling for industrial ride-hailing dispatch by combining tool-augmented global knowledge mining with utility-aligned profile exploration, resulting in significant performance gains in DiDi's production matching system.

Original authors: Tengfei Lyu, Zirui Yuan, Xu Liu, Kai Wan, Zihao Lu, Li Ma, Hao Liu

Published 2026-06-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tengfei Lyu, Zirui Yuan, Xu Liu, Kai Wan, Zihao Lu, Li Ma, Hao Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a massive, bustling city where millions of people are trying to get rides, and thousands of drivers are looking for passengers. The "brain" of this city is a super-fast computer system called a dispatcher. Its job is to match the right driver to the right passenger in the blink of an eye (literally, within 2 seconds).

For years, this dispatcher has been like a very smart accountant. It looks at numbers: How far is the driver? How much is the fare? How long has the passenger waited? Based on these hard numbers, it makes matches.

But the authors of this paper, ProfiLLM, realized the accountant was missing something important: the human story.

The Problem: The Accountant vs. The Storyteller

Imagine a driver who hates driving to the airport because it's always stuck in traffic, or a passenger who will cancel a ride if the pickup spot is more than 5 minutes away. An accountant sees "Distance: 2 miles." A storyteller sees "This driver avoids airports; this passenger hates long walks."

The researchers tried using Large Language Models (LLMs)—the same kind of AI that writes poems and answers questions—to read the history of every driver and passenger and write a "story profile" about them. They hoped these stories would help the dispatcher make better matches.

But there were three huge problems:

Too Much Data: There are millions of users. You can't ask an AI to read the entire history of every single person in real-time; it would take too long and crash the system.
The "Long Tail" Issue: Most people (like 96% of passengers) only order a few rides a month. There isn't enough history to write a good story for them.
Pretty but Useless: An AI can write a very fluent, beautiful story about a user that sounds great but doesn't actually help predict if they will cancel a ride. It's like writing a poem about a driver that sounds nice but doesn't tell you if they will actually show up.

The Solution: The "ProfiLLM" Factory

The team built a system called ProfiLLM that acts like a high-tech factory. Instead of trying to write a story for every single person in real-time, they do the heavy lifting offline (when the system is resting) and only use the results online (when the system is running).

Here is how the factory works, using a simple analogy:

Step 1: The "Global Detective" (Mining Knowledge)

First, the AI acts as a detective. It doesn't just look at one person; it looks at the entire city's history using 27 different "tools" (like a magnifying glass, a map, and a calculator).

It discovers patterns: "On rainy Tuesdays, drivers in the north district cancel more often."
It creates rules to group people together. Instead of treating 100,000 passengers as 100,000 unique individuals, it groups them into about 100 "tribes" or clusters based on similar behaviors.
Analogy: Instead of interviewing every single citizen of a country, the detective interviews the leaders of 100 neighborhoods to understand the general mood of each area.

Step 2: The "Story Editor" (Utility-Aligned Exploration)

Now, for each of these 100 "tribes," the AI tries to write a profile. But it doesn't just write one and hope it's good.

It writes many different drafts of the profile.
It runs a quick test: "If we use this story to predict behavior, does it work better than just using numbers?"
If a story is just "fluent" but doesn't help predictions, it gets thrown out. If a story helps predict cancellations or acceptances, it gets kept.
It then uses a special training technique (called DPO) to teach the AI to write only the kinds of stories that actually help the dispatcher, not just the ones that sound pretty.

Step 3: The "Speedy Matchmaker" (Online Serving)

This is the magic part. When a real passenger requests a ride:

The system does not ask the AI to think or write anything. That would be too slow.
Instead, it looks up which "tribe" (cluster) the passenger belongs to.
It grabs a pre-written, pre-tested "profile card" for that tribe that was made earlier.
It combines this card with the numbers (distance, price) and instantly makes a match.

The Results: Does it Work?

The researchers tested this on Didi, one of the world's largest ride-hailing companies.

Better Predictions: The system became much better at guessing if a driver would accept a ride or if a passenger would cancel. In tests, it improved prediction accuracy by up to 6.14%.
More Money & Happy Riders: In a simulation, the system generated 4.35% more revenue (GMV). In a real-world test over 14 days, it actually increased revenue by 0.47% and completed 0.33% more rides.
Fewer Cancellations: It reduced the number of passengers canceling before a driver even accepted by 0.82%.
Super Fast: Because the AI does all its thinking before the rush hour starts, the actual matching process adds almost zero delay (less than a millisecond).

The Bottom Line

ProfiLLM is like hiring a team of expert biographers to write detailed character profiles for every neighborhood in a city, but doing it all at night. During the day, the dispatcher just carries a pocket-sized card with the neighborhood's "personality" to make instant, smarter decisions.

It bridges the gap between the "cold numbers" of traditional systems and the "warm context" of human behavior, proving that AI can be useful in real-time industrial systems if you keep the heavy thinking offline and the execution lightning fast.

Technical Summary: ProfiLLM

1. Problem Formulation

The paper addresses the challenge of integrating Large Language Models (LLMs) into industrial ride-hailing dispatch systems to improve order–driver matching. Current production pipelines rely heavily on structured numerical features (e.g., distance, price) to predict behavioral outcomes (acceptance, cancellation, completion) for candidate order–driver (OD) pairs. However, decisive behavioral signals—such as a driver's habitual aversion to specific regions or a passenger's time sensitivity—are inherently contextual and semantic, making them difficult to capture with handcrafted numerical features.

While LLMs can distill complex behavioral trajectories into semantic profiles, deploying them in a live, millisecond-latency dispatcher faces three intertwined constraints:

Data Scale vs. Context Limits: Platform-scale logs (e.g., 44.3 million records in a single city over 38 days) exceed the context windows of any LLM.
Long-Tail Data Distribution: The majority of users (96% of passengers in the study) are low-frequency, lacking sufficient interaction history for reliable per-user profiling.
Utility Misalignment: Fluent, surface-level profiles generated by LLMs do not necessarily improve downstream prediction utility; in some cases, naive LLM profiles degrade model performance (e.g., up to -7.57% AUC drop).

The core problem is to operationalize utility-aligned user profiling that scales to millions of users, handles sparse data, and integrates into a strict offline-online contract where online inference adds sub-millisecond overhead with zero online LLM calls.

2. Methodology: ProfiLLM

ProfiLLM is an agentic LLM data pipeline designed to bridge the gap between semantic profiling and real-time matching. It enforces a strict separation: all LLM reasoning occurs offline, while the online system serves only pre-computed cluster-level profile embeddings. The framework consists of two primary modules:

2.1 Tool-Augmented Global Knowledge Mining (Offline)

To address the scale and grounding challenges, an LLM agent is equipped with a toolkit of 27 analytical tools (categorized into statistical, spatiotemporal, contextual, and causal tools). The agent follows an Explore → Deepen → Validate → Synthesize paradigm to process massive historical logs:

Explore: Computes platform-level metrics and identifies preliminary patterns.
Deepen: Conducts focused analyses on temporal patterns, spatial heterogeneity, and user segmentation.
Validate: Applies statistical hypothesis testing and causal analysis to retain only significant findings.
Synthesize: Produces three reusable artifacts:
1. Global Knowledge ( $K$ ): Platform benchmarks, temporal regularities, and causal relationships.
2. Adaptive User Clustering Rules ( $A$ ): Interpretable, rule-based classifiers that group users by behavioral similarity (e.g., active hours, frequent regions). This addresses the long-tail issue by aggregating data at the cluster level rather than the individual level.
3. Regional Priors ( $R$ ): Grid-level supply-demand statistics.

2.2 Utility-Aligned Profile Exploration (Offline)

For each user cluster, the system generates and refines profiles based on the mined global knowledge:

Candidate Generation: The LLM generates $K$ diverse candidate profiles per cluster, structured into Analysis (behavioral patterns), Profile (semantic description), and Logic (executable Boolean decision rules).
Utility Evaluation: Instead of training full prediction models for every candidate, the system uses the Logic component as a lightweight proxy. It blends the base model's predictions with rule-based predictions to compute a fused AUC. This identifies profiles that capture signals invisible to structured features.
Iterative Refinement: The best candidates are refined iteratively based on prediction error feedback.
DPO Fine-Tuning: Preference pairs are constructed by comparing profiles with different utility gains. A Direct Preference Optimization (DPO) fine-tuning step aligns the LLM generator with downstream prediction utility, allowing a compact model (e.g., Qwen3-8B) to outperform larger, unaligned backbones.

2.3 Online Serving

The online dispatcher operates without LLM inference:

Cluster Assignment: A deterministic rule evaluation assigns a user to a cluster.
Embedding Lookup: The system retrieves the pre-computed $d$ -dimensional embedding of the cluster's profile.
Prediction: The embedding is concatenated with structured features and fed into the existing multi-task prediction model.
Matching: Weights are computed, and the Kuhn-Munkres algorithm solves the assignment.
The total overhead is sub-millisecond (< 0.01 ms per pair).

3. Key Contributions

First Deployed LLM Profiling Pipeline: ProfiLLM is the first system to deploy LLM-based user profiling in a production ride-hailing dispatcher, adhering to strict latency constraints via an offline-online decoupling strategy.
Tool-Augmented Knowledge Mining: A novel module where an LLM agent autonomously composes chains of 27 analytical tools to extract global knowledge and clustering rules from data volumes exceeding LLM context limits.
Utility-Aligned Exploration: A mechanism that treats prediction utility as a first-class optimization signal. It uses a lightweight logic-based proxy and DPO fine-tuning to ensure generated profiles improve downstream matching rather than just being linguistically fluent.
Scalable Long-Tail Handling: By shifting from per-user to cluster-level profiling, the system effectively profiles low-frequency users who lack sufficient individual history.

4. Experimental Results

The system was evaluated on DiDi's production data across three Brazilian cities (City A, B, C) and deployed in a 14-day online A/B test in City A.

Prediction Performance: ProfiLLM achieved up to +6.14% relative AUC improvement in outcome prediction (specifically for passenger cancellation) compared to structured-only baselines. Naive LLM baselines often degraded performance, highlighting the necessity of the utility-alignment mechanism.
Simulation Results: In dispatching simulations, ProfiLLM yielded up to +4.35% Gross Merchandise Value (GMV) gain and significant improvements in completion rates.
Online A/B Test (City A): Over 14 days, the system demonstrated consistent real-world improvements:
- +0.47% GMV.
- +0.33% Completion Rate.
- -0.82% Cancel-Before-Accept rate.
- Reductions in post-acceptance cancellations for both drivers and passengers.
Cost Efficiency: The DPO-based refresh strategy reduced offline costs by 10.6× compared to initial exploration runs. The online path adds negligible cost, placing ProfiLLM-DPO at the Pareto frontier of cost-quality trade-offs compared to larger LLM backbones.

5. Significance and Claims

The paper claims that ProfiLLM successfully operationalizes LLMs for industrial decision-making by solving the "context window," "long-tail," and "utility alignment" trilemma. Its significance lies in demonstrating that:

Semantic Profiling is Viable at Scale: LLM-generated profiles can capture behavioral signals (like regional aversion) that structured features miss, provided they are grounded in global knowledge and aggregated at the cluster level.
Utility Alignment is Critical: Surface fluency is not a proxy for utility; optimizing profiles directly for downstream prediction metrics (via logic proxies and DPO) is essential to avoid performance degradation.
Offline-Online Decoupling is Necessary: Real-time industrial applications can leverage complex LLM reasoning only if the inference is strictly confined to offline batch processing, serving only lightweight embeddings online.

The authors position ProfiLLM not just as a theoretical improvement but as a practical, deployed solution that enhances revenue and user experience in a high-stakes, low-latency environment without modifying the core matching algorithms.