💬 NLP

Quecto-V1: Empirical Analysis of 8-bit Quantized Small Language Models for On-Device Legal Retrieval

This paper introduces Quecto-V1, a 124-million-parameter Small Language Model trained exclusively on Indian statutes and compressed to under 150 MB via 8-bit quantization, demonstrating that domain-specific, offline deployment can effectively democratize legal intelligence while preserving data sovereignty and maintaining high retrieval accuracy.

Original authors: Subrit Dikshit

Published 2026-02-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Subrit Dikshit

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

🏛️ The Problem: The "Giant Library" vs. The "Pocket Guide"

Imagine you are a lawyer or a law student in a small town in India. You need to find a specific rule in the Indian Penal Code (like "What is the punishment for theft?").

Currently, the "Smart Legal Assistants" available are like massive, cloud-based super-computers.

The Issue: They are huge, expensive, and require a super-fast internet connection to talk to.
The Risk: To use them, you have to send your private client details to a giant server in the cloud. This feels like shouting your client's secrets across a crowded room; it's risky and violates privacy.
The Barrier: Most local lawyers can't afford the expensive computers needed to run these giant models.

The paper argues that we don't need a "Super-Brain" that knows everything about the world (like cooking, space, and pop culture) to answer legal questions. We just need a specialized brain that knows only the law, fits in a pocket, and works offline.

🚀 The Solution: Quecto-V1 (The "Legal Specialist")

The authors built Quecto-V1, a tiny, super-focused AI model. Here is how they made it work, using simple analogies:

1. The "Blank Slate" Approach (Training from Scratch)

Most AI models are like students who read the entire internet before learning law. They know a lot, but they might get confused by all that extra noise.

Quecto-V1's Strategy: The authors took a blank notebook (a basic computer architecture called GPT-2) and only fed it Indian laws (The Constitution, IPC, CrPC).
The Analogy: Imagine a student who ignores all the news, sports, and movies and spends 100% of their time memorizing the Indian Penal Code. They aren't smart about everything, but they are a master of the law.

2. The "Compression" Trick (8-bit Quantization)

Usually, these AI models are huge files (like a 500MB video file). You can't run them on a regular laptop or phone.

The Fix: The team used a technique called 8-bit Quantization.
The Analogy: Think of the model as a high-resolution photo.
- Original: A 4K photo that takes up 1GB of space.
- Quecto-V1: They "compressed" it like turning that 4K photo into a high-quality JPEG. It looks almost exactly the same to the human eye, but it's 74% smaller.
- Result: The model shrunk from nearly 500MB to under 150MB. That's small enough to fit on a basic laptop or even a phone, and it runs offline (no internet needed).

📊 The Results: Small but Mighty

The authors tested Quecto-V1 against two other models:

GPT-2 (Base): A generic model with no legal training.
TinyLlama: A slightly bigger, general-purpose model.

The Race:

The Generic Models: They got confused. They tried to guess the answer based on general knowledge and often made things up (hallucinations). They were slow and needed big computers.
Quecto-V1: Because it only knows the law, it found the exact answer 88.2% of the time.
The Winner: Even though Quecto-V1 is 10 times smaller than the other models, it beat them by a huge margin on legal questions.

The Trade-off:
When they compressed the model (Quantization), it got slightly less accurate (dropped by only 2.5%), but it became 2.5 times faster and took up 73% less space.

Analogy: It's like trading a slow, gas-guzzling limousine for a nimble, electric scooter. You lose a little bit of luxury, but you gain speed, efficiency, and you can drive it anywhere without needing a gas station (internet).

🔮 What's Next? (Future Plans)

The authors admit Quecto-V1 is a "proof of concept" and has some limits. Here is their roadmap:

The "Citation" Upgrade (RAG): Currently, the AI relies on its memory. If it forgets a detail, it might guess. They plan to give it a "cheat sheet" (a database of real laws) so it can look up the exact text before answering.
Learning from the Big Boss (Distillation): They want to teach Quecto-V1 by having a giant, smart AI (like Llama-3) explain complex legal logic to it, so the small model gets smarter without getting bigger.
Speaking Local Languages: Right now, it only speaks English. They want to teach it Hindi, Tamil, and Marathi so it can help more people in India.
Case Law: Currently, it knows the rules (statutes). They want to teach it the history (court judgments) so it understands how judges have interpreted those rules in the past.

💡 The Big Takeaway

"You don't need a giant brain to do a specific job."

For specialized fields like law, a small, focused, and private AI that runs on your own computer is often better than a massive, cloud-based AI that costs a fortune and risks your privacy. Quecto-V1 proves that with the right training and compression, we can bring powerful legal intelligence to everyone, even without high-speed internet.

1. Problem Statement

The paper addresses a "Trilemma" currently facing the Legal AI sector, particularly in resource-constrained environments like India:

Accessibility Gap: State-of-the-art legal intelligence relies on massive models (7B+ parameters) and cloud APIs, making them inaccessible to users with low bandwidth or those in remote regions.
Privacy Paradox: Legal queries contain sensitive client data. Uploading this to third-party cloud providers violates client-attorney privilege and data sovereignty norms.
Compute Barrier: High-performance inference requires enterprise-grade GPUs (e.g., NVIDIA A100s), which are cost-prohibitive for individual lawyers, small firms, and educational institutions.

Existing solutions either lack generative capabilities (discriminative BERT models) or suffer from hallucinations and high resource demands (large generalist LLMs).

2. Methodology

The authors propose Quecto-V1, a domain-specific Small Language Model (SLM) designed for offline, on-device legal retrieval.

A. Model Architecture & Training Strategy

Architecture: A Decoder-Only Transformer based on the GPT-2 Small configuration (124 million parameters, 12 layers, 768 hidden dimensions).
Training Approach: Unlike standard fine-tuning, Quecto-V1 was trained from scratch with random weight initialization (Gaussian distribution). This ensures the model's reasoning is derived solely from the curated legal corpus, eliminating residual knowledge from general web-scale datasets.
Corpus: The training data consists exclusively of Indian statutes, including the Indian Penal Code (IPC), Code of Criminal Procedure (CrPC), and the Constitution of India. The goal was to maximize "lexical density" within the legal domain rather than broad world knowledge.
Hyperparameters:
- Optimizer: AdamW (decoupled weight decay) to ensure stable learning of specialized vocabulary.
- Scheduler: Linear Warmup (10%) followed by Cosine Decay.
- Precision: Mixed Precision Training (FP16) on a single NVIDIA T4 GPU to optimize memory and speed.

B. Quantization Protocol

To enable deployment on consumer-grade CPUs (laptops/mobiles), the model was converted to the GGUF format using 8-bit Symmetric Quantization (Q8_0) via llama.cpp.

Mathematical Basis: Weights are mapped from 32-bit floating-point ( $W_f$ ) to 8-bit signed integers ( $W_q$ ) using block-wise scaling: $W_q = \text{round}(W_f / \delta)$ .
Rationale: Q8_0 was chosen over aggressive 4-bit quantization to maintain a "Pareto Optimal" balance between compression and reasoning fidelity, which is critical for small models.

3. Key Contributions

Democratization of Legal AI: Demonstrated that a functional legal retrieval system can run entirely offline on standard hardware with a footprint of <150 MB.
Domain-Specific Efficiency: Proved that for narrow, high-stakes tasks (statutory retrieval), a specialized 124M parameter model outperforms generalist models with 10x more parameters.
Quantization Validation: Provided empirical evidence that 8-bit quantization reduces model size by ~74% with negligible degradation in accuracy for domain-specific tasks.
Privacy-Preserving Framework: Offered a viable alternative to cloud-based legal AI, ensuring data sovereignty by keeping all processing local.

4. Results & Comparative Analysis

Performance vs. Baselines

Quecto-V1 was benchmarked against vanilla GPT-2 (Base) and TinyLlama-1.1B on Indian Legal Definition retrieval tasks:

Model	Parameters	Training Domain	Legal Accuracy	Hallucination Rate	Memory Footprint
GPT-2 (Base)	124M	WebText (General)	12.5%	High (>60%)	500 MB
TinyLlama	1.1B	General	45.0%	Moderate (30%)	2,200 MB
Quecto-V1	124M	Indian Legal	88.2%	Low (<15%)	130 MB

Key Finding: Despite having 10x fewer parameters than TinyLlama, Quecto-V1 outperformed it by +43.2% in accuracy, validating that Domain Specificity > Model Scale for narrow tasks.

Impact of Quantization (Ablation Study)

Comparing the Full Precision (FP32) checkpoint against the Quantized (Q8_0) version:

Metric	FP32 (Original)	Q8_0 (Quantized)	Delta
Size	498 MB	132 MB	-73.5%
Inference Speed (CPU)	45 ms/token	18 ms/token	2.5x Faster
Perplexity (PPL)	18.42	18.95	+0.53 (Negligible)
Exact Match Score	91.0%	88.5%	-2.5% (Degradation)

The study confirms that 8-bit quantization offers massive gains in speed and storage with only a 2.5% drop in exact match accuracy, making it ideal for edge deployment.

5. Significance & Future Work

Significance:
The paper challenges the prevailing "Scale is All You Need" hypothesis in AI. It demonstrates that for specialized, high-stakes domains like law, a combination of domain-specific training and aggressive quantization creates a viable, privacy-preserving alternative to monolithic cloud models. This is particularly transformative for the Indian legal context, where connectivity and hardware resources are often limited.

Future Directions:

RAG Integration: Implementing Retrieval-Augmented Generation with vector databases (FAISS/ChromaDB) to reduce hallucinations by fetching real-time citations.
Knowledge Distillation: Training a "Student" model (Quecto-V2) using a larger "Teacher" model (e.g., Llama-3-8B) to transfer reasoning capabilities without increasing inference costs.
Multilingual Support: Expanding training to Hindi, Tamil, and Marathi legal corpora to serve a broader demographic.
Case Law Expansion: Incorporating Supreme Court judgments to enable precedent-based reasoning.

Limitations:
Quecto-V1 is strictly a retrieval and completion engine for its training corpus. It lacks the "world knowledge" of larger models and is not a substitute for professional legal counsel.