Architecture-Sensitive Supervised Fine-Tuning for Screen-Conditioned Action Prediction: A PiSAR Benchmark

Technical Summary: Architecture-Sensitive Supervised Fine-Tuning for Screen-Conditioned Action Prediction

Problem Statement
The paper addresses the challenge of predicting user behavior and rationale in screen-conditioned scenarios (e.g., mobile app usage). Specifically, it investigates whether frontier zero-shot Vision-Language Models (VLMs) can plausibly describe a user's intent and reasoning given a screenshot and a persona profile, or if task-specific Supervised Fine-Tuning (SFT) is required to achieve high behavioral fidelity. The authors posit that while frontier models possess broad pretraining and multimodal grounding, they may lack the specific "voice" and reasoning patterns required for precise simulation of real-world shopping and app-interaction behaviors.

Methodology
The study introduces PiSAR (Persona, Intent, Screen, Action, Rationale), a proprietary corpus of 12,929 tuples curated from public app-store reviews, Pew American Trends Panel demographics, and the OPeRA shopper traces. The data construction follows a "real on 2 of 3 slots" rule, ensuring at least two of the three core components (screen, persona, reasoning) are observed directly from real humans rather than synthesized.

The experimental setup involves:

1. Dataset Split: A 661-row held-out slice of PiSAR serves as the evaluation benchmark.
2. Models:
   • Frontier Zero-Shot Baselines: Claude Opus 4.7 and GPT-5.5, evaluated without fine-tuning.
   • Fine-Tuned Models: Two models were fine-tuned using a managed LoRA (Low-Rank Adaptation) SFT pipeline on Fireworks:
     • Qwen3-VL-8B-Instruct: Fine-tuned on two data mixes (OPeRA-only: 4,014 rows; Combined: 13,796 rows).
     • Gemma-4-26B-A4B-IT (MoE): Fine-tuned on the Combined dataset (13,796 rows) using the same hyperparameters as the Qwen runs.
3. Evaluation Metrics: Performance is measured on the same 661 rows using:
   • Semantic Similarity (sem_sim): Cosine similarity between text embeddings of predicted and gold rationales (primary metric).
   • Token Jaccard: Lexical overlap.
   • Length Ratio: Diagnostic for verbosity.
   • Threshold Pass Rates: Percentage of rows exceeding specific sem_sim thresholds (0.3, 0.5, 0.7).

Key Contributions

1. The PiSAR Benchmark: A standardized, screen-anchored evaluation slice for behavioral rationale prediction, allowing direct comparison between frontier zero-shot models and fine-tuned agents.
2. Architecture-Sensitive SFT Findings: The paper demonstrates that the efficacy of SFT is highly dependent on the base model's post-training prior. A specific fine-tuning recipe (LoRA rank 16, 13k examples) yielded a massive performance lift on an 8B instruct-tuned model (Qwen) but failed to move a 26B reasoning-tuned model (Gemma) beyond the frontier zero-shot baseline.
3. Recipe-vs-Model Mismatch Diagnosis: The authors identify that reasoning-tuned models with "draft-first" or chain-of-thought post-training priors resist displacement by standard SFT recipes, leading to "template bleed" (e.g., outputting internal reasoning markers instead of the final rationale) and poor performance on direct-output tasks.

Results

• SFT vs. Frontier: The fine-tuned Qwen3-VL-8B-Instruct (Combined) achieved a sem_sim of 0.783, significantly outperforming the frontier zero-shot baselines (Claude Opus 4.7: 0.459; GPT-5.5: 0.482). This represents an absolute gap of 0.30.
• Strict Paraphrase Performance: On the strict threshold of sem_sim ≥ 0.7, the fine-tuned Qwen cleared 79% of test rows, whereas the frontier baselines cleared only 1–2%.
• Base Model Sensitivity: The same training data and recipe applied to Gemma-4-26B-A4B-IT resulted in a sem_sim of 0.441, placing it in the same performance band as the frontier zero-shot baselines.
• Failure Mode Analysis: The Gemma model exhibited "reasoning-trace bleed," where 4% of outputs began with internal markdown markers (e.g., * Persona:, * Draft 1:) and failed to produce a final answer, leading to extreme length ratios (mean 4.43, max 457x) and higher latency.
• Data Composition: Increasing the training data from OPeRA-only (4k rows) to Combined (13k rows) improved the Qwen model's performance from 0.519 to 0.783, with the largest gains observed on app-store data.

Significance and Claims
The paper claims that for screen-conditioned behavioral simulation, domain-relevant fine-tuning of a small, direct-output vision-language base is decisively superior to prompting frontier reasoning models.

• Efficiency: The fine-tuned 8B model operates at sub-second latency (0.79s median) and at a fraction of the inference cost of frontier models, while achieving significantly higher behavioral fidelity.
• Architectural Fit: The results suggest that "reasoning-tuned" bases (like Gemma-4-IT) may require stronger fine-tuning methods (e.g., higher LoRA rank, full-parameter FT) or more data to override their post-training priors. Conversely, models with direct-output priors (like Qwen3-VL-Instruct) are highly receptive to standard SFT recipes.
• Reproducibility: While the PiSAR corpus and weights are proprietary, the authors provide the full methodology, data fusion rules, and training configurations, asserting that the "shape" of the bet (10–20k real-world records + 8B base + LoRA SFT) is the reproducible key to success, rather than waiting for larger frontier models.

The authors explicitly state they do not claim fine-tuning is universally better than prompting, nor that Gemma is a "bad" model. Instead, they argue that on this specific task, the interaction between the SFT recipe and the base model's prior is the critical variable, and that a mismatch here can render a 26B reasoning model less effective than a 0.30-absolute-gain 8B fine-tune.