Kathleen: Oscillator-Based Byte-Level Text Classification Without Tokenization or Attention

Kathleen is a highly efficient, parameter-minimal text classification architecture that achieves state-of-the-art performance on standard benchmarks by processing raw UTF-8 bytes directly in the frequency domain using novel components like RecurrentOscillatorBanks and PhaseHarmonics, thereby eliminating the need for tokenization, attention mechanisms, and large embedding tables.

The Gist

Imagine you are trying to understand a book. Most modern AI models (like the famous "Transformers") work like a librarian who first has to break the book down into individual words, look up each word in a massive dictionary to understand its meaning, and then try to figure out how those words relate to each other. This process is powerful, but it's slow, requires a huge amount of memory, and breaks if the book gets too long.

Kathleen is a new kind of AI that skips the dictionary entirely. Instead of reading words, it listens to the raw "sound" of the text.

Here is the story of how Kathleen works, explained through simple analogies:

1. The Problem: The "Word-First" Bottleneck

Think of a standard AI model as a translator who only speaks "Word Language." Before it can understand a sentence, it must translate every letter into a word.

• The Issue: If you give it a very long document (like a whole novel), the translator gets overwhelmed. The memory needed to hold all those word connections grows so fast (like a snowball rolling down a hill) that the computer crashes.
• The Fix: Kathleen doesn't translate. It treats the text like a music signal. It looks at the raw stream of bytes (the digital "notes") without worrying about what the words mean yet.

2. The Core Idea: The "Resonant Tuning Forks"

Kathleen's brain is built on a concept called Resonance.

• The Analogy: Imagine a room full of different tuning forks. If you hum a specific note, only the fork tuned to that note will start vibrating loudly. The others stay silent.
• How Kathleen uses it: Instead of looking for words, Kathleen has thousands of tiny "digital tuning forks" (called Oscillator Banks). When text flows through it, these forks vibrate if they detect specific patterns (like the rhythm of a sentence or the frequency of certain letters).
• The Benefit: This is incredibly fast. While other models try to compare every word to every other word (which is slow), Kathleen just listens for the "vibrations" as the text passes by. It's like listening to a song once versus trying to write down every note and compare them later.

3. The Secret Sauce: The "Magic 6-Parameter Knob"

The paper discovered something surprising. They built a huge, complex machine with millions of parts, only to realize that most of it was unnecessary.

• The Analogy: Imagine a high-end stereo system with 50 knobs. You turn them all, and the music sounds okay. Then, you realize that if you just tweak one single tiny screw on the volume dial, the sound becomes perfect.
• The Reality: The most important part of Kathleen is a component called PhaseHarmonics. It has only 6 learnable numbers (parameters).
  • Removing a massive, complex "bio-inspired" brain section (560,000 parts) only hurt performance by a tiny bit.
  • Removing those 6 tiny numbers crashed the performance by a huge amount.
  • Lesson: Sometimes, simple mathematical tricks work better than complex, human-like "thinking" structures.

4. The "FFT-Rotate" Encoder: The Universal Translator

Usually, computers need a massive table (like a phone book) to remember what every letter or byte means. This takes up a lot of space.

• Kathleen's Trick: Instead of a phone book, Kathleen uses a mathematical magic trick (FFT-Rotate). It takes a single, tiny vector of numbers and spins it around to create a unique "signature" for every single byte (0–255).
• The Result: It replaces a massive dictionary (65,000 numbers) with a tiny key (256 numbers) that works just as well, or even better.

5. Why This Matters: The "Long-Document" Superpower

Because Kathleen listens to the "sound" of the text rather than mapping word-to-word connections, it doesn't get tired.

• The Analogy: A standard AI is like a person trying to hold hands with everyone in a stadium; if the stadium gets too big, the line breaks. Kathleen is like a radio wave; it can travel across the entire stadium without breaking.
• The Result: Kathleen can read a 100,000-byte document (a whole book chapter) on a standard computer chip. A standard AI would run out of memory after just a few pages.

Summary: The "Kathleen" Effect

• No Dictionary: It reads raw bytes, not words.
• No "Attention": It doesn't stare at every word to see how they relate; it listens for patterns.
• Tiny Size: It is 16 times smaller than similar models but often smarter at understanding text.
• The Magic: It proves that you don't need a giant, complex brain to understand language. Sometimes, you just need a few well-tuned "tuning forks" and a good ear for the rhythm of the data.

In short, Kathleen is the AI that realized: "We don't need to know every word to understand the song; we just need to hear the melody."

Technical Summary

1. Problem Statement

Modern Natural Language Processing (NLP) is dominated by Transformer-based models, which face three critical limitations:

1. Quadratic Complexity: The $O(L^2)$ attention mechanism limits scalability, causing GPU memory exhaustion on long sequences (e.g., full documents).
2. Tokenizer Dependency: Reliance on tokenizers introduces language-specific preprocessing, lossy compression, and engineering complexity (e.g., out-of-vocabulary issues).
3. Parameter Inefficiency: Competitive performance typically requires millions to billions of parameters.

These issues are exacerbated in byte-level processing, where input sequences are 3–5× longer than tokenized equivalents. Existing byte-level models (e.g., ByT5, CANINE) still rely on Transformers and attention, inheriting the $O(L^2)$ bottleneck. The authors ask: Can frequency-domain processing on raw bytes match or exceed tokenized models without attention, using orders of magnitude fewer parameters?

2. Methodology: The Kathleen Architecture

Kathleen is a text classification architecture that operates directly on raw UTF-8 bytes using frequency-domain processing. It replaces attention mechanisms and tokenizers with a pipeline of signal-processing inspired components. The final model, Kathleen-Clean, contains only 733K parameters.

Core Components

The architecture processes bytes through the following pipeline:

1. FFT-Rotate Wavetable Encoder:

   • Problem Solved: Replaces the massive embedding table ($256 \times d$ parameters) with a single learnable vector ($d$ parameters).
   • Mechanism: It maps all 256 byte values using a single learnable vector $w$ and an FFT-based phase rotation: $Enc(b) = \mathcal{F}^{-1}[\mathcal{F}[w] \odot e^{i \cdot b \cdot 2\pi/255}]$.
   • Efficiency: Reduces encoder parameters from ~65K to 256 floats while improving accuracy.

2. RecurrentOscillatorBank:

   • Mechanism: Uses causal convolution kernels initialized as damped sinusoids ($k_i(t) = \gamma_i^t \cos(\omega_i t)$).
   • Function: Acts as a bank of resonators that selectively amplify specific frequency patterns in the byte stream while attenuating noise.
   • Memory: Augmented with a recurrent memory state ($M_t$) to accumulate evidence across the sequence, enabling $O(L)$ processing.

3. PhaseHarmonics (The MVP):

   • Mechanism: A sinusoidal non-linearity that concatenates the input with projections at exponentially spaced frequencies: $PH(x) = [x, \sin(x \cdot 2^0 + \phi_0), \dots, \sin(x \cdot 2^5 + \phi_5)]$.
   • Parameters: Only 6 learnable phase offsets ($\phi_0 \dots \phi_5$).
   • Impact: Despite its tiny size, it creates multi-resolution spectral views of the data.

4. PowerLawGate (PLG):

   • Mechanism: Applies a power-law non-linearity $sign(x) \cdot |x|^\gamma$ (where $\gamma \approx 0.5$), mimicking the Weber–Fechner law in psychophysics.
   • Function: Compresses dynamic range to prevent high-amplitude patterns from dominating.
   • Context: Proven effective only in frequency-domain contexts (not in tokenized word embeddings).

5. DualPooling:

   • Combines attention-weighted pooling and max pooling to reduce sequences to vectors, crucial for preserving sparse informative signals in short texts.

3. Key Contributions & Ablation Insights

The paper emphasizes an ablation-driven design process, starting from a 1.8M parameter predecessor and systematically removing components to find the optimal architecture.

• PhaseHarmonics is the Single Most Impactful Component: Removing this 6-parameter module caused a 2.6% drop in accuracy. Conversely, removing a massive 560K parameter "bio-inspired" cognitive framework (Phantasy) caused only a 0.2% drop.
• Frequency vs. Cognitive: The study demonstrates that simple frequency-domain components systematically outperform complex, bio-inspired cognitive architectures for this task.
• Context-Dependent Utility: The PowerLawGate was useless in tokenized contexts (0.0% gain) but contributed +0.9% in byte-level frequency contexts, proving that architectural utility depends on the input representation.
• Carrier Cancellation Discovery: Early experiments using sinusoidal carriers failed because mean pooling destroyed the signal ($E[\sin(\omega t)] \approx 0$). The solution was to use identity-preserving frequency features (Fourier encoding) rather than explicit carriers.

4. Experimental Results

Kathleen-Clean was evaluated on three standard benchmarks (IMDB, AG News, SST-2) against tokenized models and other byte-level Transformers.

Metric	Kathleen-Clean (Ours)	Tokenized Kathleen (11.8M params)	CANINE-S (132M params)
IMDB Accuracy	88.6% (+1.6% vs Tok)	87.0%	N/A
AG News Accuracy	92.3% (+2.1% vs Tok)	90.2%	N/A
SST-2 Accuracy	83.3%	N/A	85.8%
Parameters	733K	11.8M	132M
Complexity	$O(L)$	$O(L)$	$O(L^2)$

• Parameter Efficiency: Kathleen-Clean is 16× smaller than its tokenized counterpart and 180× smaller than CANINE-S, yet outperforms them on IMDB and AG News.
• Long-Context Scalability: Due to $O(L)$ complexity, Kathleen can process sequences up to 100K+ bytes. In contrast, standard Transformers run out of GPU memory (OOM) beyond 1,024 bytes.
• Gap to Pretrained Models: There remains an ~8% accuracy gap compared to BERT (93% on SST-2), attributed to BERT's massive pretraining on external corpora and larger parameter count.

5. Significance and Future Work

• New Pareto Frontier: Kathleen establishes a new efficiency frontier for NLP, proving that frequency-based signal processing is a viable alternative to attention for text understanding.
• Edge Deployment: With only 733K parameters, the model is small enough to run on microcontrollers (e.g., ESP32) and mobile devices.
• Streaming & Long-Context: The $O(L)$ complexity enables real-time streaming classification and the processing of entire documents without memory constraints.
• Language Agnosticism: By operating on raw bytes, the model requires no tokenizer training and handles any language naturally.

Conclusion: Kathleen demonstrates that "less is more" in NLP architecture. By leveraging the mathematical structure of frequency processing and oscillators, it achieves state-of-the-art efficiency, outperforming much larger tokenized models while eliminating the need for tokenizers and attention mechanisms.