💬 NLP

POLARIS: Guiding Small Models to Write Long Stories

The paper introduces POLARIS, a low-compute GRPO training recipe that combines LLM-as-a-judge rewards with human-reference injection to enable small models like Qwen3.5-9B to generate high-quality, long-form stories that rival much larger models while maintaining strong length adherence and generalization.

Original authors: Rishanth Rajendhran, Jenna Russell, Mohit Iyyer, John Frederick Wieting

Published 2026-06-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Rishanth Rajendhran, Jenna Russell, Mohit Iyyer, John Frederick Wieting

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: Small Writers Get Tired

Imagine you have a very smart but small robot writer (a "small model"). If you ask it to write a short story, it's great. But if you ask it to write a long novel, two bad things happen:

It gives up: It stops writing way before the story is finished.
It loses its mind: The story gets messy, repetitive, or boring as it gets longer.

Big, expensive super-computers (like "frontier models") can write long stories well, but they are too costly for most people to use. The authors of this paper asked: "Can we teach a small, cheap robot to write long, high-quality stories without needing a supercomputer?"

The Solution: POLARIS

They created a training recipe called POLARIS. Think of this as a special coaching method for the robot. Instead of just letting the robot practice alone, they added two secret ingredients to its training camp.

Ingredient 1: The "Strict Editor" (The LLM Judge)

Usually, when training AI, you use a simple "scorecard" that just says "Good" or "Bad." That's like a teacher giving a student a grade of "C" without telling them why.

POLARIS uses a Frontier LLM Judge (a very advanced AI) acting as a Strict Editor.

How it works: Instead of just giving a score, this Editor reads the story and fills out a detailed Rubric (a checklist).
The Checklist: It looks at 16 different things, like "Did the character have a clear voice?" (Good) or "Did the story get repetitive?" (Bad).
The Analogy: Imagine a writing workshop where a famous author reads your story and gives you specific notes: "Your dialogue is too stiff," or "The ending feels rushed." The robot learns from these specific notes, not just a generic grade.

Ingredient 2: The "Human Anchor" (Human-Reference Injection)

This is the most unique part. In a typical training group, the robot generates 5 different stories, and the computer picks the best one to learn from. But sometimes, all 5 stories are mediocre, and the robot gets stuck in a loop of writing "okay" but not "great" stories.

POLARIS adds a Human Anchor to every group.

How it works: For every prompt, the system forces the robot to look at a real human-written story alongside its own attempts.
The Analogy: Imagine a group of students trying to solve a math problem. Usually, they only compare their answers to each other. But in this class, the teacher puts a perfectly solved example on the board. The students aren't just guessing; they have a "North Star" to aim for. Even if the robot doesn't copy the human story, seeing that high-quality example keeps the robot's "ambition" high and prevents it from settling for low-quality writing.

The Results: The "Small" Model That Punches Up

The authors took a standard 9-billion-parameter model (Qwen3.5-9B) and trained it using this method on about 1,400 short stories.

The Surprise: Even though the robot was only trained on stories up to 4,000 words, it learned how to write stories up to 12,000 words (3 times longer!) without losing quality.
The Comparison:
- Vs. Base Model: It is much better than the untrained version.
- Vs. Giant Models: It performs almost as well as models that are 3 times larger (27 billion parameters) and much more expensive.
- The "Stress Test": Most small models collapse when asked to write long stories. POLARIS is the only small model that stayed strong, keeping the story coherent and actually finishing the length requested.

Why This Matters

The paper argues that "Length Generalization" (the ability to write longer than you were trained to) is a great stress test.

The Metaphor: If you train a runner to run a 1-mile race, they might get tired at 2 miles. If they can run 3 miles without stopping, it proves they have true stamina, not just a short burst of speed.
POLARIS proved that with the right "coaching" (the Strict Editor and the Human Anchor), a small model can have the stamina of a giant one.

Summary

POLARIS is a clever, low-cost training method that teaches small AI models to write long, creative stories by:

Giving them detailed, specific feedback from a smart AI editor.
Showing them high-quality human examples to keep them aiming high.

The result is a small, affordable AI that writes long stories as well as the massive, expensive ones, without needing a supercomputer to run it.

Technical Summary: POLARIS – Guiding Small Models to Write Long Stories

Problem Statement

Small open-weight language models (e.g., 9B parameters) struggle significantly with long-form creative writing. When tasked with generating stories of substantial length, these models typically exhibit one of two failure modes: they either fall far short of the requested length, or their output quality degrades precipitously as the generation extends. This stands in contrast to frontier models, which maintain coherence and stylistic consistency over thousands of tokens. Existing reinforcement learning (RL) approaches to long-form generation often rely on expensive infrastructure, including large base models, continual pretraining, custom-trained reward models, or massive datasets. The paper asks whether a small open model can become competitive in this domain without such scale.

Methodology

The authors present POLARIS (Policy Optimization with LLM-as-a-judge rewards and Anchored-Reference Injection for Storywriting), a lower-compute GRPO (Group Relative Policy Optimization) recipe designed specifically for long-form creative writing. The method is applied to a Qwen3.5-9B base model using a dataset of approximately 1,388 prompt–story pairs derived from 100 commercially purchased short-story anthologies.

The recipe relies on two primary technical innovations:

1. Structured LLM-as-a-Judge Reward

Instead of training a separate scalar reward model, POLARIS employs a frontier LLM (Gemini 3 Flash) as an online judge during each GRPO step. This judge utilizes a structured Story Quality rubric comprising 16 dimensions:

Positive Dimensions (6): Prompt fulfillment, narrative arc, character depth, voice, scene realization, and thematic richness.
Negative Dimensions (10): Prompt violations, coherence breaks, generic language, over-summary, over-explanation, structural drift, dialogue problems, mechanical errors, predictability, and overwrought prose.
Catch-alls: Bonus and penalty terms for unmodeled excellence or failure.

The raw score is normalized and combined with auxiliary penalties for repetition, length mismatch, and blank outputs. Crucially, the reward function includes a sublinear length-scaling factor to prevent reward inflation on short outputs and a clipping mechanism to stabilize training.

2. Human-Reference Injection (HRI)

To address the issue of stagnation in open-ended RL—where policy rollouts may converge to similar reward scores, halting learning—the authors introduce HRI.

Mechanism: For each GRPO group, a human-written story (teacher-forced through the current policy) is inserted as the $(k+1)$ -th member.
Function: This human reference acts as a high-reward anchor, maintaining within-group reward variance and providing a stable gradient pressure toward stronger writing.
Implementation: The reference is scored by the same reward function as policy rollouts but is excluded from the calculation of group mean and standard deviation. Its advantage is scaled by a warmup schedule to dampen the gradient contribution from the demonstration trajectory, ensuring the update remains a biased demonstration-augmented GRPO step rather than an unbiased on-policy estimator.

Key Contributions

A Low-Compute Recipe: The authors demonstrate that a 9B model can be trained for approximately 48 hours on 4 A100 GPUs (total cost ~$500) to achieve competitive long-form writing performance.
Structured Rubric over Scalar Rewards: By replacing trained reward models with a structured, multi-dimensional LLM-as-a-judge, the method avoids the need for separate reward-model training pipelines and provides granular feedback on specific narrative qualities.
Human-Reference Injection: The introduction of HRI as a mechanism to prevent learning stagnation in open-ended tasks, offering a simple yet effective way to anchor policy updates without requiring full expert trajectories for every step.
Length Generalization: The training data consists entirely of stories up to 4,000 words. Despite this, the resulting model, POLARIS–9B, demonstrates robust performance on prompts requesting stories up to 12,000 words (3× the training length).

Results

The evaluation spans five benchmarks covering in-distribution (ID) and out-of-distribution (OOD) prompts and rubrics, involving over 160,000 pairwise judgments.

Performance vs. Larger Models: POLARIS–9B is competitive with open-weight models three times its size (e.g., Qwen3.5-27B, Gemma 4 31B). On EQ-Bench Creative, it ranks 3rd among 15 models, behind only frontier models (GPT-5.4, Claude Opus 4.6) and human references.
Length Adherence: Unlike many larger open-weight models that either degrade in quality or undershoot requested lengths at 8–12k words, POLARIS–9B maintains both rubric quality and length adherence. It achieves a length adherence ratio of 0.72 at 8–12k words while preserving story quality, whereas baselines like Gemma 4 31B often write significantly shorter stories to maintain quality.
Human Evaluation: In a blinded human evaluation of 60 prompt–story pairs, annotators preferred POLARIS–9B over the base Qwen3.5-9B (67.5% win rate). The comparison with Qwen3.5-27B was statistically indistinguishable (51.2% win rate), with human comments highlighting POLARIS–9B's stronger atmosphere, voice, and scene realization.
Failure Modes: The paper notes that while POLARIS–9B avoids the "quality collapse" and "length runaway" seen in other models, it can still suffer from stylistic overloading (overuse of jargon or heavy prose) and local coherence failures (contradicting details).

Significance and Claims

The paper claims that length generalization serves as a meaningful stress test for creative-writing models. A model that merely learns surface-level patterns will degrade as stories require sustained narrative coherence and arc completion over thousands of tokens. POLARIS–9B's ability to retain strong rubric scores at 8–12k words, despite training only on data up to 4k words, suggests that the proposed GRPO recipe with HRI and structured judging is an effective path for aligning small models on open-ended tasks with soft rewards.

The authors position this work as a practical alternative to the "scale-first" approach, showing that with the right training recipe, small open-weight models can rival much larger counterparts in long-form creative writing without the need for massive compute infrastructure or proprietary data. The results suggest that distinguishing close models in this domain requires evaluating not just aggregate quality, but specifically how quality and length adherence hold up under far-transfer conditions.