Just-In-Time Objectives: A General Approach for Specialized AI Interactions

This paper introduces "Just-In-Time Objectives," a framework that passively observes user behavior to infer and rapidly optimize for specific, real-time goals, enabling large language models to generate specialized tools and responses that significantly outperform standard generic interactions.

The Gist

Here is an explanation of the paper "Just-In-Time Objectives," using simple language and creative analogies.

The Big Problem: The "Generic" AI

Imagine you hire a very smart, well-traveled chef (the AI) to cook you dinner. You say, "Make me something good."

The chef, wanting to be safe and please everyone, makes a giant, bland salad. It's edible, and technically "good," but it's not what you actually wanted. You wanted a spicy curry because you're feeling adventurous, or a warm soup because you're sick. But since you didn't specify, the chef just gives you the default menu item.

This is the current state of Large Language Models (LLMs). They are trained to be helpful to everyone, so they default to being generic, safe, and boring. They don't know your specific mood, your specific project, or the specific problem you are trying to solve right now.

The Solution: "Just-In-Time" Objectives

The authors of this paper propose a new way to talk to AI. Instead of giving the AI a vague order, they let the AI watch what you are doing and figure out your goal for you, right in the moment.

They call this "Just-In-Time Objectives" (JIT).

Think of it like a GPS navigation system:

• Old Way: You tell the GPS, "Take me to a good place." The GPS picks a random tourist spot.
• JIT Way: The GPS sees you are driving fast, checking your watch, and heading toward the highway. It realizes, "Ah, this person is in a rush to get to an airport." It instantly switches its goal to: "Get to the airport in the fastest possible time."

The AI doesn't need you to type a long, perfect prompt. It just needs to see what you are doing to understand what you need.

How It Works: The "Poppins" System

The researchers built a tool called Poppins (named after Mary Poppins, who always had the perfect tool for the job in her magical bag).

Here is the step-by-step process of how Poppins works:

1. The Observation (The Eyes):
   You are working on a document or a design on your computer. Poppins (a browser extension) quietly looks at your screen. It sees you are editing a research paper about "AI systems."

2. The Inference (The Brain):
   Instead of waiting for you to ask, Poppins thinks: "Okay, this person is writing a technical section. They probably need help making the complex ideas clearer, or maybe they need a diagram."
   It creates a Just-In-Time Objective.

   • Generic Objective: "Help with writing."
   • JIT Objective: "Clarify the technical architecture of the AI system for a computer science audience."

3. The Action (The Hands):
   Poppins uses this specific objective to build a custom tool just for you.

   • It doesn't just give you text. It might build a drag-and-drop diagram tool to help you visualize your system.
   • It might summon a "Technical Writing Expert" persona that gives you feedback specifically on how to explain complex code.
   • It might create a "Character Emotion Tracker" if you are writing a sci-fi story.

Why Is This Better? (The Results)

The researchers tested this with real people doing real work (writing papers, coding, planning trips).

• The "Generic" AI: Gave helpful but boring advice like "Check your spelling" or "Make sentences shorter."
• The "JIT" AI: Gave specific, high-quality help like "Your diagram is missing a feedback loop between the user and the server," or "Here is a custom tool to track your character's emotions."

The Stats:

• People preferred the JIT AI 66% to 86% of the time over the standard AI.
• Users felt the JIT AI understood their specific needs much better.
• The tools generated were unique to every single person. One person got a "Neural Architecture Explorer," while another got a "Cultural Perspective Highlighter."

The Analogy: The Tailor vs. The Department Store

• Standard AI is like a Department Store. It sells one-size-fits-all suits. They fit "okay," but they aren't perfect for your body type or your specific style.
• Just-In-Time Objectives are like a Master Tailor. The tailor watches you move, sees your posture, and notices you are going to a wedding. They instantly cut the fabric to fit your specific body and the specific event.

The Catch (Limitations)

• It takes a moment: The AI needs a few seconds to "watch" and "think" before it builds the tool. It's not instant like a quick chat message.
• Privacy: To work, the AI has to look at your screen. The researchers are very careful about this, but it means you have to trust the system with what you are doing.
• Control: Sometimes the AI guesses wrong. But the cool part is that the "Objective" is visible. You can see what the AI thinks you want, and you can click a button to say, "No, actually, I want to focus on the design, not the code."

The Bottom Line

This paper suggests that the future of AI isn't about us learning how to talk to robots better. It's about robots learning how to read us better.

By watching what we do and figuring out our goals on the fly, AI can stop being a generic chatbot and start being a personalized assistant that builds exactly the tools we need, exactly when we need them.

Technical Summary

Here is a detailed technical summary of the paper "Just-In-Time Objectives: A General Approach for Specialized AI Interactions."

1. Problem Statement

Large Language Models (LLMs) are powerful but often produce generic, safe, and homogenized outputs because their training objectives are fixed in advance to satisfy a broad, undefined audience. This creates a "Gulf of Envisioning" where users struggle to articulate specific, momentary goals via natural language prompts.

• The Core Issue: Standard LLMs optimize for a "committee" of objectives (reasoning, safety, general helpfulness), leading to a convergence on mediocre, non-opinionated advice.
• User Limitation: Users often cannot precisely define their immediate needs (e.g., "clarify the logic of this specific paragraph for a CHI reviewer") or lack the cognitive bandwidth to craft complex prompts for every micro-task.
• Goal: The paper proposes a method to move from static, implicit objectives to dynamic, explicit, user-specific objectives induced in real-time ("Just-In-Time") to specialize AI behavior for the specific task at hand.

2. Methodology: Just-In-Time (JIT) Objectives Architecture

The authors propose a generalizable architecture that shifts the burden of objective definition from the user to the system, which infers goals from passive observation.

A. Core Components

The architecture operates on a Generate-Evaluate loop, enhanced by a JIT Objective layer:

1. Objective Induction:
   • Input: Passive observation of user context (screenshots, text content, file attachments, browser logs).
   • Process: A Vision Language Model (VLM) analyzes the context to infer the user's immediate goals.
   • Output: A structured JSON specification containing:
     • name: A concise label (e.g., "Strengthen narrative argument").
     • description: 1-2 sentences detailing the goal.
     • weight: An estimated importance score (1–10).
2. Generation (gen_objective):
   • The induced objective is prepended to the prompt of a generative LLM.
   • This steers the model to produce artifacts (text, code, designs) specifically aligned with the inferred goal rather than generic best practices.
3. Evaluation (eval_objective):
   • The induced objective is used as a rubric for an LLM-as-a-judge evaluator.
   • Instead of generic quality scoring, the evaluator ranks candidates based on how well they satisfy the specific JIT objective (e.g., filtering out feedback that is technically correct but narratively weak).

B. System Implementation: Poppins

The authors instantiated this architecture in Poppins, a browser extension and web application.

• Input Pipeline: Captures the user's screen (screenshot + extracted text) to infer context.
• Two Modes of Assistance:
  1. Poppins-experts (Expertise Generation): Generates "expert personas" (e.g., "HCI Researcher," "Technical Writer") with specific backgrounds and methodologies. It then generates feedback or brainstorming ideas from these personas, filtered by the JIT objective.
  2. Poppins-tools (Tool Generation): Designs and synthesizes interactive software tools (e.g., a drag-and-drop diagram builder, a character emotion tracker) tailored to the user's specific task. It generates the UI code (Svelte components) and implements the tool logic.
• User Agency: Users can view, edit, select, or delete the induced objectives, experts, and tool designs before final generation, providing a "human-in-the-loop" check.

3. Key Contributions

1. Generalizable Architecture: A framework for automatically inducing user-specific objectives from interaction traces to steer LLMs without manual prompting.
2. Poppins System: A working prototype demonstrating JIT objectives in two domains:
   • Generating specialized expert feedback.
   • Generating functional, interactive software tools (UI generation) on the fly.
3. Objective as a First-Class Object: Treating the AI's goal as a visible, modifiable, and steerable interface element, bridging the gap between user intent and model execution.

4. Evaluation Results

The paper presents three studies involving 236 total participants across different scales.

Study 1: Accuracy & Utility (N=14, 70 inputs)

• Objective Quality: Induced objectives were rated highly for accuracy (M=2.04/3) and usefulness (M=2.18/3) on a Likert scale. 75% were rated "Accurate" or "Very Accurate."
• Preference: Outputs generated with JIT objectives won 71–86% of pairwise comparisons against a baseline LLM (same model, no JIT objective) for expertise, tool designs, and feedback.

Study 2: Generalizability (N=205, 410 inputs)

• Objective Selection: In 97.8% of cases, participants preferred the induced JIT objective over writing a custom one, indicating high relevance.
• Performance: JIT objectives maintained strong win rates (66–71%) over baseline LLMs across diverse tasks (creative writing, coding, planning).
• Best-of-N Scaling: Using a JIT evaluator to select the best candidate from a pool of 100 samples significantly improved win rates (up to 75% in Study 1), proving the evaluator effectively filters for user-aligned outputs.

Study 3: In-Lab Usability (N=17, 1-hour sessions)

• Relevance: 94.1% of participants rated Poppins-experts' objectives as "relevant" or "very relevant."
• Quality: Poppins-experts achieved significantly higher overall quality ratings (p < .05) compared to a standard LLM chat interface.
• User Experience: Participants reported that JIT objectives reduced the cognitive load of prompt engineering, allowing them to focus on high-level task refinement rather than low-level wording.
• Tool Generation: Users were impressed by the ability to generate custom tools (e.g., "Neural Architecture Search Explorer") that they had not previously considered, though some preferred textual feedback over full UI generation.

5. Significance and Implications

• Breaking the "Generic" Trap: JIT objectives demonstrate that even generic foundation models can be coerced into highly specialized, divergent, and opinionated behaviors by dynamically changing their optimization target.
• Shift in HCI Paradigm: Moves the interaction model from Prompt Engineering (user defining the goal) to Objective Induction (system inferring the goal, user refining it). This lowers the barrier to entry for complex AI interactions.
• Generative UIs: Validates the feasibility of generating not just text, but functional, interactive software tools tailored to specific user contexts, moving beyond static chat interfaces.
• Ethical Considerations: The paper highlights risks regarding privacy (screen monitoring), attribution (fake experts), and agency (users accepting induced goals without scrutiny), calling for transparent interfaces and user controls.

In summary, the paper argues that by observing users and inducing Just-In-Time objectives, AI systems can transition from being generic assistants to specialized, on-demand collaborators that generate both content and the tools necessary to create it.