🤖 machine learning

Reflector: Arrangement-Aware Harmonic Retrieval for Sample-Based Composition

This paper presents Reflector, an open-source interactive audio workstation that dynamically adapts sample retrieval to a composer's evolving harmonic context by using a synthetic-trained embedding space to approximate a hand-designed interval-class oracle, thereby enabling real-time, arrangement-aware harmonic discovery without relying on copyrighted training data.

Original authors: Austin Rockman

Published 2026-07-27

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Austin Rockman

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Sound of a Growing Song

Imagine you are a music producer trying to build a track. You have a library of thousands of sound clips—drums, synths, vocal chops, and weird noises. Usually, when you want to find a new sound to add, you pick one clip you like and ask a computer, "What sounds like this?" The computer then shows you a list of sounds that are very similar in tone or texture. This is like asking a librarian, "Do you have any books that look exactly like this one?" It's helpful, but it doesn't help you write a new story.

Music, however, isn't just about finding things that look alike; it's about how things fit together when they play at the same time. This is the science of harmony. Think of harmony like a puzzle where pieces only click if their shapes match, not just if they are the same color. If you have a sad, slow piano note, you might want a bright, happy drum beat to go with it, even though they sound totally different. The challenge for computers is that as you build a song, the "puzzle" changes. Every time you add a new layer, the rules for what fits next shift. A sound that was perfect for your first drum might clash with your third. This paper explores a new way to help composers navigate this shifting landscape, turning a static search into a living, breathing assistant that understands the whole song as it grows.

Reflector: The Song That Remembers

Meet Reflector, a new tool designed to help music composers find the perfect sounds for their songs. But unlike other tools that just look for sounds that resemble your starting point, Reflector is obsessed with what sounds compatible with the whole song you are building right now.

Think of composing a song like building a house with a team of friends. In the old way, you'd pick a brick (a sound clip) and ask, "What other bricks look like this one?" You'd get a pile of identical bricks. But Reflector asks a different question: "We are building a wall right now. We have a red brick here and a blue brick there. What kind of new brick would fit perfectly in the gap between them?"

The Magic "Oracle"

At the heart of Reflector is a secret recipe called an oracle. Imagine a giant, hand-drawn chart that acts like a music judge. This chart doesn't care if two sounds are the same; it cares about the "distance" between their notes. It has a rulebook that says, "If you play a C and an E together, that's a happy, strong combination. If you play a C and a F-sharp, that's a tense, clashing combination."

The researchers built this rulebook by hand, assigning scores to every possible pair of musical notes. They call this the "interval-class oracle." It's like a strict teacher who knows exactly which notes get along and which ones fight.

The Problem with the Strict Teacher

Here is the twist: If you ask this strict teacher to pick the best sound to go with any song, it gets lazy. It realizes that a very quiet, simple sound (like a soft hum) fits with almost everything. So, if you just used the teacher's rulebook directly, it would keep suggesting that same quiet hum over and over again, no matter what you were building. It would ignore the interesting, complex sounds because they are "riskier."

The paper found that using the rulebook directly is like having a recommendation engine that only suggests plain white bread because it goes with everything. It's safe, but it's boring.

The Smart Student (The AI)

To fix this, the researchers trained a smart computer student (a neural network) to learn from the strict teacher. But they didn't just tell the student to memorize the answers. They taught the student to understand the spirit of the rules.

They created thousands of fake musical examples using a computer synthesizer (no real copyrighted music was used, just math and sound waves). They showed the student pairs of sounds and the teacher's scores. The student learned to translate these sounds into a special 128-dimensional map (a giant, invisible playground). In this map, sounds that fit well together are close to each other, and sounds that clash are far apart.

The magic is that this student learned to be a better judge than the teacher. While the teacher would always pick the boring "universal" sound, the student learned to spread things out. It realized that a good song needs variety. So, when you ask the student for a match, it doesn't just give you the safest option; it gives you a diverse list of sounds that actually create interesting musical relationships.

The Song That Grows With You

The coolest part of Reflector is how it handles a song as you build it. Imagine you are arranging a song on a timeline with many tracks.

The Live Centroid: As you drag and drop sounds onto your timeline, Reflector constantly calculates a "center point" for your current song. It's like a magnetic core that moves as you add new layers.
The Sweep-Line: The system scans your timeline to see which sounds are playing at the same time. It figures out the "harmonic identity" of that specific moment.
The Shift: If you change a note or add a new instrument, the "center point" of your song moves. Reflector instantly updates its suggestions. If you suddenly switch from a happy major key to a spooky minor key, the tool stops suggesting happy sounds and starts looking for spooky ones, even if you started the search with a happy sound.

The paper tested this on a library of 631 samples. They found that when they used the old "direct rule" method, the system only ever suggested 64 different sounds out of the 631, and one boring sound appeared in 97% of the top results. But when they used the trained student (the embedding), the system successfully suggested 625 of the 631 sounds. It covered the whole library while still knowing which sounds fit together.

A Galaxy of Songs

Reflector also lets you look at your entire history of songs. It takes every song you've made and turns it into a single dot in a 3D space. If you have two songs that feel harmonically similar, their dots will be close together. You can fly through this "galaxy" of your own work, finding connections between songs you made months apart. It helps you see how your musical style is evolving.

What It Can't Do

The authors are honest about the limits. Reflector is great at harmony (how notes fit), but it doesn't care about the texture or rhythm of the sound. It might suggest a sound that fits the notes perfectly but sounds like a squeaky toy when you need a drum. It also only understands Western music scales (the kind used in most pop and classical music). If you are making music with different musical rules, the tool might not work as well.

The Big Takeaway

The main discovery of this paper is that you can't just use a simple rulebook to find musical matches; it gets stuck on boring, safe answers. But if you train a computer to learn the geometry of those rules, you get a system that is both smart and diverse. Reflector shows that by letting the computer understand the whole song as it changes, we can turn a static search tool into a creative partner that grows with the artist.

The entire system runs on a single computer, uses no copyrighted data to learn, and is free for anyone to use. It proves that with the right design, computers can help us navigate the complex, shifting world of making music, not just by finding copies, but by finding the perfect fit.

Technical Summary: Reflector – Arrangement-Aware Harmonic Retrieval for Sample-Based Composition

Problem Statement

Existing music information retrieval (MIR) tools for sample-based composition typically rely on static queries, such as a single reference loop or a rendered mixdown. While effective for initial ideation, these systems become less informative as an arrangement evolves. As composers place new sounds on a timeline, the harmonic context shifts, rendering a fixed reference query obsolete. Current commercial solutions (e.g., Splice Create, Output Co-Producer) address this by suggesting complementary sounds in real-time but operate on proprietary, unpublished methodologies over their own catalogs. There is a lack of published, open-source systems that model a multi-track arrangement as an evolving set of co-sounding regions to drive harmonic retrieval, nor is there a framework for how such representations persist across sessions to record compositional practice.

Methodology

Reflector is an interactive audio workstation designed to track harmonic combinations as they accumulate on a timeline and adapt retrieval accordingly. The system operates through three primary architectural components:

1. The Compatibility Oracle

At the core of the system is a hand-designed "oracle": a fixed $12 \times 12$ table of interval weights ( $K$ ) that scores how pitch-class content interacts between two sources.

Frame-level Scoring: For two chroma frames $c$ and $d$ , the score is a kernel-weighted bilinear form: $s_{frame} = \frac{c^\top K d}{(1^\top c)(1^\top d) + \epsilon}$ . This normalizes for loudness, focusing on the average interval quality of the combination.
Trajectory-level Scoring: Scores aggregate over time, weighted by "interaction mass" (the product of activation masses of both sources). This ensures that moments where both sources are silent do not dilute the score.
Transposition Search: The oracle evaluates candidates under all 12 circular shifts of the pitch-class axis, emitting a full 12-score profile to identify the optimal transposition for blending.

2. Synthetic Training Pipeline

To avoid copyright issues and gain full control over the training distribution, the system is trained entirely on synthetic audio.

Data Generation: Two stages generate paired chroma trajectories. Stage A constructs pitch-class sets from six triadic shapes with stochastic extensions and temporal patterns (sustained chords, arpeggios, etc.). Stage B renders these into audio via additive synthesis with randomized domain parameters (waveform, reverb, distortion, etc.).
Architecture: A two-tower shared encoder (3x Conv1D, BatchNorm, GELU, pooling) maps context and candidate chroma trajectories into a 128-dimensional embedding space.
Loss Functions:
- Stage A: Supervised by Mean Squared Error (MSE) against the oracle's best transposition score and Forward KL divergence against the full 12-way transposition profile.
- Stage B (Fine-tuning): Uses InfoNCE loss to optimize the dot-product geometry directly on audio-derived chroma, with a small MSE anchor to prevent drift from the oracle's judgments.
Key Invariance: The system learns transposition invariance through data augmentation (random circular shifts) and the oracle's profile supervision, rather than hard-coding it.

3. System Process and Session Analysis

Indexing: Samples are segmented into overlapping 3-second windows, embedded, and weighted by their mean pairwise oracle coherence to form a file-level embedding.
Session Representation: As a composer arranges material, a "sweep-line" analysis identifies co-sounding regions. These regions are aggregated into a "session centroid" (a weighted mean of region centroids), which updates dynamically as material is added, removed, or transposed.
Navigation: Session centroids are projected into a navigable 3-D space ("Galaxy"), allowing composers to visualize structural harmonic relations and navigate between sessions based on harmonic vocabulary.

Key Findings and Results

The paper characterizes the system's behavior on a working library of 631 samples, yielding several critical insights:

Degeneracy of Direct Scoring: When the oracle kernel is used directly as a retrieval rule (without the learned embedding), retrieval collapses onto a few "universal donors" (e.g., a sparse, consonant drone). Because the kernel judges pairs in isolation and lacks a concept of variety, it favors samples that blend acceptably with nearly everything. Adjusting kernel weights does not resolve this.
Embedding as a Filter: The learned 128-dimensional embedding preserves the oracle's pairwise judgments (achieving an NDCG of 0.85 at rank 10) while covering the entire library (625 of 631 samples appear in top-10 lists). The paper attributes this to the normalized geometry of the embedding space: on a unit hypersphere, no single candidate can be close to every query direction simultaneously. The contrastive training objective (InfoNCE) enforces uniformity, effectively filtering out the "degenerate" solutions that direct scoring favors.
Distinct Retrieval Surfaces: The paper identifies two distinct retrieval surfaces:
- Similarity (Cosine): Retrieves siblings and versions (what resembles the query).
- Compatibility (Embedding): Retrieves complements (what combines with the query).
  These surfaces are nearly disjoint, with an anti-correlation of $\rho = -0.65$ . Reflector exposes both, allowing composers to toggle between finding "versions" and finding "harmonies."
Arrangement Awareness: Retrieval is not static. As a session evolves (e.g., through a tritone pivot), the session centroid shifts, and the top-10 suggestions change significantly (70% turnover in the cited example). The system tracks the composer's harmonic drift across sessions, revealing relationships that a single static query would miss.

Significance and Claims

The paper positions Reflector as a "systems account" rather than a claim of superior musicality. Its primary contributions are:

Open-Source Architecture: It provides a public, end-to-end design for an arrangement-aware retrieval system that runs locally on a single machine, contrasting with proprietary commercial platforms.
Ethical Training: The entire pipeline is trained on synthetic audio with no copyrighted data, addressing legal and ethical uncertainties in music AI.
Distillation of Rules: It demonstrates that a fixed, hand-designed harmonic rule (the oracle) can be distilled into a neural embedding that retains the rule's interaction logic while overcoming its retrieval limitations (degeneracy) through geometric constraints.
Compositional Visibility: The system makes the "latent compositional structure" of a sample library visible, tracking how a composer's harmonic practice evolves over time and across sessions.

The authors explicitly state that the kernel weights are an "experimental design position" and not necessarily perceptually optimal. The system's validation is presented as its utility in practice ("validation is use") rather than through a perceptual study, which is identified as a necessary future step. The system currently focuses on Western tonal harmony and pitch-class relationships, leaving timbral and rhythmic judgments to the composer.