Semantic Prompting: Agentic Incremental Narrative Refinement through Spatial Semantic Interaction

1. Problem Statement

The paper addresses the limitations of current Large Language Model (LLM) systems in supporting sensemaking, a non-linear cognitive process where users iteratively organize fragmented information into coherent narratives. Existing "Space-to-Text" approaches suffer from three critical usability gaps:

1. Interaction-Revision Misalignment: Current methods (e.g., regeneration-from-scratch) often regenerate entire reports in response to minor spatial changes. This causes high linguistic variance and unintended rewording, forcing users to manually verify unaffected sections rather than focusing on refinements.
2. Human-LLM Intent Misalignment: The "Black Box" nature of LLMs creates a transparency gap. Users cannot see why the model made specific changes, leading to a disconnect between the user's spatial intent (e.g., highlighting a keyword) and the model's output (e.g., inserting the keyword in the wrong context).
3. Inflexible Granular Customization: Existing systems rely on rigid templates or strict object-to-text mappings. They fail to support nuanced, incremental refinements where users need to adjust specific details or structures without overhauling the entire narrative.

2. Methodology: Semantic Prompting & S-PRISM

To address these gaps, the authors propose Semantic Prompting, a framework that translates spatial semantic interactions into targeted, executable prompts for LLMs. They implemented this framework in a system called S-PRISM (Semantic Prompting and Refinement for Interactive Spatial Manipulation).

Core Architecture: Multi-Agent Pipeline

S-PRISM utilizes a hierarchical multi-agent pipeline divided into four stages:

1. Interact: Users manipulate a spatial workspace (using frames, highlights, sticky notes, and document clustering) to organize information.
2. Perceive: The system captures two types of data:
   • Prompt Adjustments: Changes to task descriptions or output formats.
   • Semantic Interactions: 15 types of spatial actions (e.g., adding/removing frames, editing highlights, repositioning notes).
3. Reason: A specialized Inferring Agent analyzes the spatial interactions to deduce human intent. It answers: "Why did the user perform this interaction?" and "How should the report be revised?" Crucially, this agent surfaces its reasoning to the user via interface bubbles to ensure transparency.
4. Act: A Refining Agent executes the narrative updates based on the inferred intent. It adheres to strict heuristics:
   • Minimize unnecessary rephrasing (maintain text stability).
   • Restrict edits to the user-intended sections (targeted updates).
   • Preserve structural consistency.
   • Output: The system performs a diff-comparison, highlighting only the changed text in the report panel.

Key Design Goals

• DG1: Support targeted space-text mapping (partial, incremental revisions).
• DG2: Surface LLM intent inference (transparency).
• DG3: Enable user-initiated refinement (user controls when to trigger updates).

3. Key Contributions

1. Semantic Prompting Framework: A novel interaction paradigm bridging spatial semantic interactions with LLM reasoning, enabling targeted narrative revisions through intuitive spatial metaphors rather than manual prompt engineering.
2. S-PRISM System: A functional prototype implementing a hierarchical multi-agent pipeline that manages reasoned semantic prompts and executes precise, positional refinements.
3. Empirical Validation: Quantitative evidence showing S-PRISM significantly outperforms regeneration baselines in refinement accuracy and semantic fidelity.
4. User Study Insights: Qualitative validation (N=14) demonstrating that the system effectively resolves interaction-revision misalignment and intent misalignment, fostering trust and enabling incremental formalization.

4. Results

Empirical Evaluation (Quantitative)

The system was tested against a "Regeneration-from-Scratch" baseline using a dataset of 35 interaction pairs (based on the Sign of the Crescent benchmark).

• Targeted Refinement: S-PRISM achieved a Precision of 0.951 compared to the baseline's 0.752. While the baseline had perfect recall (1.0) due to global rewriting, S-PRISM maintained high precision (0.831) by avoiding unintended changes to unaffected text.
• Semantic Fidelity: S-PRISM significantly outperformed the baseline in Precision (0.558 vs. 0.348) and F1-score (0.614 vs. 0.463), indicating that the revisions more accurately reflected the specific semantic interactions performed in the workspace.

User Study (Qualitative)

A study with 14 participants evaluated the system's efficacy in resolving the three identified gaps.

• Incremental Formalism: Users reported that S-PRISM supported steady, cumulative improvements in report correctness, whereas the baseline showed instability and plateauing performance in complex tasks.
• Intent Alignment: The "Reasoning" bubbles (showing the agent's logic) were rated highly for building trust. Participants noted that seeing the inference chain bridged the "Transparency Gap."
• Granular Customization: Users successfully used Notes as "contextual meta-prompts" to rectify errors (e.g., clarifying that a highlighted keyword belongs to a specific section) and Frames to reorganize report hierarchies dynamically.
• Zero-Prompt Success: In Phase II, four participants successfully transitioned tasks (e.g., from reports to itineraries) using only spatial manipulations without modifying textual prompts, validating the system's ability to derive intent from spatial semantics alone.

5. Significance

This paper makes a significant contribution to Human-Computer Interaction (HCI) and AI-assisted writing by:

• Shifting the Paradigm: Moving away from "Regeneration" (replacing the whole) to "Refinement" (targeted editing), which aligns better with the iterative nature of human sensemaking.
• Enhancing Trust: By making the LLM's reasoning process visible and editable, the system mitigates the "Black Box" problem, allowing users to verify and correct the AI's intent before finalizing changes.
• Bridging Modalities: It successfully translates non-verbal spatial actions (clustering, highlighting) into complex linguistic instructions, reducing the cognitive load of prompt engineering.
• Future Direction: The work suggests a path toward "Goal-level" alignment and mixed-initiative systems where AI can proactively suggest refinements based on spatial conflicts, further empowering knowledge workers in complex synthesis tasks.