Fine-Tuning Causal LLMs for Text Classification: Embedding-Based vs. Instruction-Based Approaches

This paper demonstrates that for resource-constrained single-label text classification, fine-tuning causal LLMs with a classification head on final-token embeddings is significantly more parameter-efficient than instruction tuning while achieving comparable or superior performance to both instruction-tuned LLMs and domain-specific BERT models.

The Gist

Imagine you have a giant, incredibly smart library assistant (a Large Language Model, or LLM) who has read almost everything in the world. You want to hire this assistant to sort a massive pile of patent documents into specific categories. The problem? This assistant is huge, expensive to run, and usually trained to write stories, not sort files.

This paper is a guide on how to teach this giant assistant to sort files efficiently, using just one standard computer graphics card (GPU) instead of a supercomputer. The authors tested two different ways to train the assistant and found that one method is much better than the other for this specific job.

Here is the breakdown of their findings using simple analogies:

The Two Training Methods

The researchers tried two different "training camps" for the assistant:

1. The "File Folder" Method (Embedding-Based)

• How it works: Imagine you ask the assistant to read a document and then hand you a single, perfect summary note written on the last page. You then attach a small, simple label maker (a "classification head") to that note to decide which folder the document goes into.
• The trick: They didn't retrain the whole assistant. They just taught the assistant how to write that one perfect summary note and how to use the label maker. They used a technique called "LoRA" (Low-Rank Adaptation), which is like giving the assistant a set of sticky notes to write on instead of rewriting their entire brain.
• Result: This method was incredibly fast, cheap, and accurate. It used very few "trainable" resources (like a small budget) but got the job done perfectly.

2. The "Chatbot" Method (Instruction-Based)

• How it works: Instead of asking for a summary note, you talk to the assistant like a chatbot. You say, "Here is a document. Please tell me what category it belongs to." The assistant then has to type out the answer word by word.
• The trick: This requires the assistant to learn how to follow instructions and generate text in a specific format.
• Result: This method was slower and required a much larger budget (more "trainable" resources) to get good results. It worked okay for complex tasks with many categories, but it was often picky about how you asked the question. If the prompt was slightly off, the assistant might get confused or write extra words that broke the system.

The Big Showdown: What They Found

The authors tested these methods on patent data (legal documents about inventions) and compared them to older, smaller models (like BERT) that were built specifically for sorting tasks.

• For Single-Label Sorting (One category per document):
  The "File Folder" method won hands down. It matched or even beat the older, specialized models and the "Chatbot" method, but it did so while using 10 to 30 times fewer resources. It was like using a Swiss Army knife to cut a steak: it worked just as well as a chef's knife but was much lighter and cheaper to carry.

• For Multi-Label Sorting (Multiple categories per document):
  The "Chatbot" method had a slight edge, but only if you were willing to spend a lot more money on training (using a huge budget of resources). Even then, the "File Folder" method was still very competitive.

• Speed and Efficiency:
  The "File Folder" method was much faster at both training and running. The "Chatbot" method was slower because it had to "think" and type out the answer letter by letter, whereas the "File Folder" method just looked at the summary note and clicked a button.

The "Magic" of the Small Budget

One of the coolest findings is that you don't need a massive, expensive model to get great results.

• They used a relatively small model (3 Billion parameters) with the "File Folder" method and it beat the "Chatbot" method using a much larger model.
• They even tested the "Chatbot" method on the most expensive, state-of-the-art models available from big tech companies (like GPT-5 and Claude Opus) without training them at all. Even these super-smart, frozen models couldn't beat the small, trained "File Folder" model. It's like a well-trained local mechanic beating a brand-new, untrained Formula 1 car in a specific repair job.

The Catch (Limitations)

The paper is honest about where this method isn't perfect:

• Speed vs. Accuracy: While the "File Folder" method is great, it is still about 20 times slower than the older, specialized models (BERT) when it comes to pure speed. If you need to sort millions of documents per second, the older models are still the kings of speed.
• Statistical Confidence: The "File Folder" method was numerically better, but the difference wasn't statistically "proven" to be huge in every single test. It's consistently better, but the margin of victory is sometimes small.
• Training Instability: Sometimes, the "File Folder" method would fail to learn if the random starting point (the "seed") was unlucky, requiring the researchers to try a few times to get a good result.

The Bottom Line

If you need to sort text documents (like patents) and you have limited computer power (like a single graphics card), the best strategy is to treat the giant AI model like a feature extractor (the "File Folder" method). Don't try to make it chat or write essays; just ask it to summarize the document and attach a simple label maker. This approach is cheaper, faster, and often more accurate than trying to teach the AI to follow complex instructions or using older, specialized models.

Technical Summary

Technical Summary: Fine-Tuning Causal LLMs for Text Classification

Problem Statement
Text classification has traditionally relied on fine-tuning encoder-based transformers (e.g., BERT, RoBERTa), which utilize a special classification token (e.g., [CLS]) to aggregate sequence information. In contrast, decoder-only (causal) Large Language Models (LLMs) are pre-trained for next-token prediction with left-to-right attention, lacking an explicit classification token and bidirectional visibility over the input. While causal LLMs possess billions of parameters trained on trillions of tokens, adapting them for classification is challenging due to their size, which often renders full fine-tuning infeasible on single-GPU hardware. This paper investigates whether causal LLMs can be effectively fine-tuned for classification under resource constraints and compares two distinct adaptation strategies: embedding-based fine-tuning versus instruction-based fine-tuning.

Methodology
The authors evaluate two approaches using quantized Low-Rank Adaptation (QLoRA) to enable training on a single NVIDIA L4 GPU (24GB VRAM). All models are loaded in 4-bit precision (NF4) using the BitsAndBytes library, with only the LoRA adapters and task-specific heads updated.

1. Approach 1: Embedding-Based Fine-Tuning (Decoder Tuning)

   • Mechanism: The causal LLM acts as a feature extractor. The hidden state of the final token (which implicitly attends to all preceding tokens) is extracted as a sequence representation. A lightweight classification head (linear layer or feed-forward network) is attached to this embedding to predict class labels.
   • Training: Optimizes class posteriors directly via cross-entropy (single-label) or binary cross-entropy (multi-label). The LoRA rank ($r$) is set to 8 or 16, with a small subset of parameters (typically 5.6M–42M) updated.
   • Inference: A single forward pass yields the final token embedding, followed by a lightweight classification layer computation.

2. Approach 2: Instruction-Based Fine-Tuning

   • Mechanism: The classification task is reformulated as a prompt-response generation problem. Inputs are converted to prompts (e.g., "What is the category?"), and the model is trained to generate the label text as a response.
   • Training: Optimizes the likelihood of the generated label tokens using next-token prediction loss. This requires the model to learn specific formatting and verbalization of labels. LoRA ranks are higher ($r=64$), resulting in a larger trainable budget (45M–167M parameters).
   • Inference: Requires sequential decoding of the label tokens, which introduces latency compared to the embedding approach.

Key Contributions

• Decoder-Only Classification Strategy: Demonstrates that causal LLMs can effectively serve as classifiers by leveraging their final token embeddings as aggregate sequence representations, analogous to the [CLS] token in encoders.
• Resource-Efficient Benchmarking: Reports state-of-the-art results on patent classification tasks using single-GPU friendly methods (QLoRA + 4-bit quantization), proving that models up to 8B parameters can be fine-tuned efficiently.
• Comparative Analysis: Provides a systematic comparison showing that for single-label classification, the embedding-based approach matches or exceeds instruction-tuned performance while training 10–30× fewer parameters. Instruction tuning is found to be competitive only in multi-label regimes and only with substantially larger trainable budgets.
• Practical Guidelines: Offers empirical evidence on the trade-offs between throughput, calibration, and robustness, suggesting that embedding-based methods are more robust to prompt variations and offer better calibration than instruction-based methods.

Results
Experiments were conducted on two patent datasets: a proprietary 5-class single-label corpus (CLV) and the public WIPO-Alpha multi-label dataset (14 categories).

• Single-Label Performance: The embedding-based approach (Approach 1) consistently achieved competitive F1 scores, often surpassing instruction-tuned models (Approach 2) and domain-specific BERT baselines. For instance, a 3.2B parameter Llama-3.2 model with $r=8$ achieved an F1 of 0.860 on CLV, outperforming the best BERT baseline (0.854) while updating only ~12M parameters compared to 346M for BERT.
• Multi-Label Performance: On the WIPO dataset, Approach 2 (specifically Mistral-7B with $r=64$) achieved the highest F1 (0.819), outperforming Approach 1. However, this required 167.8M trainable parameters, negating the "parameter-efficient" advantage in this specific regime.
• Throughput: Approach 1 demonstrated significantly higher training and inference throughput (samples per second) compared to Approach 2. While Approach 1 was slower than BERT-class encoders (~20× slower), the authors note that knowledge distillation can recover BERT-class throughput with a minimal F1 cost (≤1.5 points).
• Statistical Significance: Paired McNemar tests and bootstrap $\Delta$F1 95% confidence intervals indicate that while the embedding-based approach numerically outperforms instruction tuning on single-label tasks, the difference is not statistically significant at $p<0.05$.
• External Validation: On the AG News dataset, the embedding-based approach (Llama-3.2-3B, $r=8$) achieved an F1 of 0.929, comparable to strong BERT baselines and instruction-tuned models, confirming generalization beyond the patent domain.
• Closed-Source Models: Frontier closed-source models (e.g., GPT-5, Claude Opus 4.6) used in zero-shot or few-shot prompting modes failed to match the performance of the fine-tuned 1–3B parameter Llama models using Approach 1, highlighting the necessity of supervised adaptation for high-accuracy classification.

Significance and Claims
The paper claims that parameter-efficient, embedding-based fine-tuning of causal LLMs is an effective, scalable, and high-performing alternative to both conventional BERT-style models and instruction-tuned LLMs for text classification.

• Efficiency: The study demonstrates that high-performance classification can be achieved on single-GPU hardware by freezing the base model and updating only a tiny fraction of parameters via LoRA.
• Robustness: The embedding-based approach is claimed to be more robust to prompt engineering errors and offers better-calibrated probability outputs compared to instruction-based generation, which can suffer from formatting brittleness.
• Practicality: For single-label tasks, the embedding approach is presented as the preferred strategy, offering a superior F1-to-compute trade-off. For multi-label tasks, the paper acknowledges that while instruction tuning can yield higher accuracy, it often requires parameter budgets comparable to full BERT models, thus limiting its efficiency advantage.
• Limitations: The authors modestly note that their claims are bounded by the use of proprietary data for single-label results, the lack of statistical significance in head-to-head comparisons, and the throughput penalty of LLMs compared to BERT (though mitigatable via distillation). They also highlight that training instability can occur with certain seeds, recommending multiple runs for reproducibility.

In conclusion, the work provides empirical evidence that specialized, resource-constrained fine-tuning of causal LLMs via embedding heads is a viable and often optimal path for domain-specific text classification, lowering the barrier to deploying advanced language models in specialized NLP tasks.