Open-Set Source Tracing as Compositional Factors via Structured Prototypes

Technical Summary: Open-Set Source Tracing as Compositional Factors via Structured Prototypes

Problem Statement
The paper addresses the limitations of current synthetic speech attribution (Source Tracing) tasks, which traditionally treat a "source" as a monolithic generative architecture. The authors argue that this approach is forensically insufficient because a generative model's output is shaped by a compositional tuple of factors: the Architecture (A), the Training Data (D), and a Residual Configuration (H) (encompassing hyperparameters, optimization stochasticity, and non-linear interactions).

The core challenge is that standard Deep Metric Learning methods (e.g., ArcFace) compress these factors into a single embedding, causing entanglement. This prevents models from generalizing to "Open-Set" scenarios where a known architecture is paired with unseen data, or vice versa. The paper defines four attribution scenarios ranging from Fully Closed (seen combinations) to Fully Open (both architecture and data unseen), highlighting the need for compositional generalization.

Methodology
The proposed framework redefines source tracing as a factorized attribution problem using Structured Orthonormal Prototypes and Subspace Partitioning. The methodology is built on three progressive strategies:

1. Variability Decomposition: The authors distinguish between inter-source variability (the target: $A \times D \times H$) and intra-source variability (nuisance factors like reference speakers or phonetic content). The goal is to make the model invariant to inference-time variables ($i$) while capturing the source tuple $s = (a, d, h)$.
2. Strategy 1: Structured Orthonormal Prototypes: Instead of learnable class weights, the model uses a fixed, orthonormal prototype matrix $P$. This enforces geometric regularity and maximizes inter-class separability, acting as a strong prior for open-set identification.
3. Strategy 2: Subspace Partitioning: To explicitly separate factors, the embedding space is partitioned into orthogonal subspaces:
   • $Z_A$ (Architecture): Aligned with architecture prototypes.
   • $Z_D$ (Training Data): Aligned with dataset prototypes.
   • $Z_R$ (Residual): A dedicated, prototype-free subspace capturing the residual configuration $H$ and unlabeled variability.
   • Loss Functions: The framework employs a weighted cross-entropy loss combined with Mean Squared Error (MSE) terms to align embeddings with their respective prototypes. For the residual subspace, an Energy Constraint is introduced to prevent representation collapse or norm dominance, ensuring $Z_R$ encodes bounded uncertainty without overshadowing the primary factors.

Key Contributions

• Redefinition of Source Identity: The paper formally defines a source as a compositional tuple $S = (A, D, H)$ rather than a single label, enabling the modeling of shared characteristics between models.
• Factorized Embedding Space: The introduction of subspace partitioning allows the model to learn marginal representations for architecture and data independently, facilitating generalization to novel combinations of known factors.
• Residual Modeling: The explicit inclusion of a residual subspace ($Z_R$) with energy constraints allows the system to absorb non-deterministic factors and non-linear interactions without corrupting the primary factor representations.
• Structured Prototypes: The use of frozen, orthonormal prototypes provides a geometric structure that improves few-shot performance compared to standard learnable weights.

Experimental Results
The framework was evaluated on the MLAAD dataset using a few-shot open-set identification protocol (K=1, 3, 5 support examples).

• Baseline Performance: The standard ArcFace baseline achieved high closed-set accuracy (95.11%) but suffered a significant generalization gap (13.70%) in open-set scenarios.
• Strategy 1: Improved open-set performance over the baseline but struggled with "Unseen Combinations" due to the lack of shared structure between prototypes.
• Strategy 2 (Factorized): Showed substantial improvements. Specifically, Strategy 2.B (with Residual Modeling) achieved the best overall results, reaching 84.52% F1-Macro in the "All vs All" few-shot setting (K=5).
• Generalization: Strategy 2.B significantly reduced the generalization gap (to 9.19%) compared to the baseline. It demonstrated particular strength in "Unseen Architecture" and "Unseen Dataset" scenarios, suggesting that isolating factors allows the model to leverage known data-driven artifacts to compensate for novel architectures, and vice versa.

Significance and Claims
The paper claims that decomposing the generative identity into factorized marginal representations is essential for robust open-set attribution. The authors assert that their approach:

1. Enables Compositional Generalization: The system can attribute audio from novel $(A, D)$ pairings by recombining learned factor prototypes, rather than requiring exhaustive retraining for every new source.
2. Improves Scalability: The factorized approach scales more favorably than monolithic classification; representing $C_A \times C_D$ sources requires only $C_A + C_D$ basis vectors, whereas monolithic approaches require a unique prototype for every combination.
3. Offers Interpretability: The separation of subspaces suggests a path toward forensic analysis where the contribution of architectural vs. data-driven cues can be quantified by comparing similarity within specific subspaces ($Z_A$ vs. $Z_D$).

The authors conclude that while their method significantly outperforms angular-margin baselines in few-shot open-set scenarios, the current implementation absorbs inference-time variability (like reference speakers) into the residual subspace, leaving explicit compensation for these variables as a direction for future work.