💬 NLP

Controlled Paraphrase Geometry in Sentence Embedding Space: Local Manifold Modeling and Latent Probing

This paper introduces a local manifold modeling framework and the CoPaGE-300K dataset to analyze and generate synthetic points within the sentence embedding space, demonstrating that while nonlinear geometric models accurately capture local semantic structures, their geometric validity does not necessarily translate to improved downstream classification performance.

Original authors: Leonid Bedratyuk

Published 2026-05-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Leonid Bedratyuk

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant, invisible map where every sentence in the world is a dot. This is called "sentence embedding space." If you say, "The cat sat on the mat," and someone else says, "The feline rested on the rug," those two dots land very close together on the map because they mean roughly the same thing.

For a long time, scientists assumed that if you took a bunch of sentences that mean the same thing (like a group of friends who all say "hello" in slightly different ways), they would just form a simple, flat cluster of dots—like marbles scattered on a flat table.

This paper asks a different question: Is that cluster actually flat, or is it curved, like a hill or a bowl?

Here is the breakdown of what the author, Leonid Bedratyuk, discovered, using simple analogies.

1. The Experiment: Building a "Controlled" Cloud

To test this, the researcher didn't just grab random sentences from the internet. That would be like trying to study the shape of a cloud by looking at the whole sky—it's too messy.

Instead, he built a controlled laboratory. He created a "sentence factory" using templates.

The Template: "The [Role] [Action] a [Object] for [Group]."
The Control: He swapped out the words in the brackets with synonyms (e.g., changing "Role" from "doctor" to "physician" or "Action" from "prescribed" to "recommended").

This created a "cloud" of thousands of sentences that are almost identical in meaning but slightly different in wording. He then looked at where these sentences landed on the map.

2. The Discovery: It's Not Flat; It's Curved

The researcher tried to fit a flat sheet of paper (a "linear model") over these dots. It didn't work well. The dots kept poking through the paper.

He then tried fitting a curved surface, like a bowl or a saddle (a "quadratic" or "cubic" model).

The Result: The curved surface fit the dots perfectly.
The Analogy: Imagine a flock of birds flying in a specific formation. If you try to draw a straight line through them, you miss most of them. But if you draw a smooth, curved line that follows their flight path, you hit them all. The paper proves that sentences with similar meanings don't just sit in a flat pile; they follow a specific, curved path in the computer's brain.

3. The New Tool: "Surface-Based" Generation

Once the researcher found the shape of this curved path (the "manifold"), he invented a way to create new, fake sentences that fit perfectly on that curve.

Old Way (Linear Interpolation): Imagine taking two points on a hill and drawing a straight line between them. If you walk along that straight line, you might fall off the side of the hill into the void. This is how most computers currently try to make new sentences—they just average two existing ones.
New Way (Surface-Based): The researcher's method is like a roller coaster track. He builds a track that follows the exact curve of the hill. When he generates a new point, he places it directly on the track.
- The Benefit: The new points are mathematically guaranteed to be "on the hill." They respect the natural shape of the language.

4. The Twist: Being "Geometrically Correct" Doesn't Mean Being "Useful"

This is the most surprising part of the paper. The researcher tested if these new, perfectly curved sentences helped a computer learn to classify things better (like telling if a sentence is about medicine or education).

The Expectation: "If we add more perfect examples to the training data, the computer should get smarter, right?"
The Reality: No. Adding these geometrically perfect points did not automatically make the computer's classification better.
The Analogy: Imagine you are teaching a student to recognize a specific type of tree. You show them 10 perfect photos of the tree. Then, you show them 10 more photos that are perfectly identical to the first 10, just slightly shifted. The student doesn't learn anything new because the new photos don't show any new angles or features; they just repeat what was already known.

The paper concludes that geometric validity (the point fits the curve) is different from discriminative utility (the point helps the computer make a better decision). Just because a sentence fits the mathematical shape of the language doesn't mean it adds new information to help a classifier.

5. The Gift: CoPaGE-300K

Finally, the researcher didn't just write a paper; he built a dataset called CoPaGE-300K.

Think of this as a standardized test kit for other scientists.
It contains 300,000 sentences built from his controlled templates.
It allows other researchers to test their own theories about the "shape" of language without having to build their own factories from scratch.

Summary

The Map: Sentences with similar meanings form a curved shape, not a flat pile.
The Tool: We can now build new sentences that follow this curve perfectly, like a train on a track.
The Lesson: Just because a new sentence fits the curve perfectly doesn't mean it helps a computer learn better. Sometimes, "perfect" geometry is just a repetition of what we already know.
The Resource: The author released a massive, controlled dataset so others can study these shapes further.

The paper is essentially a mapmaker saying, "I found the true shape of this territory, and I built a tool to walk along it, but walking along it doesn't always help you find the treasure."

1. Problem Statement

Sentence embeddings are ubiquitous in NLP, yet the local geometry of these spaces—specifically how controlled semantic variations (paraphrases) are organized—remains poorly understood.

The Gap: It is unclear whether semantically close sentences form simple linear (affine) clusters or complex, low-dimensional nonlinear manifolds within the high-dimensional embedding space.
The Challenge: Existing methods for data augmentation (e.g., linear interpolation, random perturbation) often ignore the underlying geometric structure of the embedding cloud, potentially generating points that are geometrically inconsistent with the local semantic patch.
The Goal: To explicitly model the local geometry of controlled semantic classes using low-degree fitted surfaces and to determine if synthetic latent points generated on these surfaces preserve geometric validity, and whether this validity translates to downstream discriminative utility.

2. Methodology

The paper proposes a surface-based latent probing procedure that operates in three main stages:

A. Data Construction (CoPaGE-300K)

To isolate local semantic variability from the noise of natural corpora, the authors introduced CoPaGE-300K, a controlled dataset containing over 300,000 sentence variants.

Structure: Sentences are generated from three template families (Institutional, Educational, Medical) with 4 lexical slots.
Regimes: Variations range from single-slot changes (C1) to full combinatorial variations of all four slots (C5), creating controlled "embedding clouds" of varying complexity.
Embeddings: All sentences are encoded using sentence-transformers/all-mpnet-base-v2 (768-dim).

B. Local Geometric Modeling

The core hypothesis is that these clouds lie on low-dimensional nonlinear manifolds.

Dimensionality Reduction: The high-dimensional cloud ( $d=768$ ) is projected into a reduced PCA space ( $r \ll d$ ) preserving 90% of the variance. This stabilizes the fitting process. The dimension $r$ is adaptive (ranging from 10 to 37 depending on complexity).
Surface Fitting: The authors fit explicit algebraic models to the reduced cloud:
- Affine (Linear): Baseline.
- Quadric (Quadratic): $f(z) = z^T Q z + L^T z + C = 0$ .
- Cubic: Used for more complex geometries.
- Fitting: Parameters are estimated by minimizing the algebraic residual of the implicit equation.

C. Surface-Based Latent Generation

Instead of linear interpolation, new synthetic points are generated via a two-step process:

Barycentric Initialization: An internal point $v$ is created as a convex combination of existing cloud points (sampled via Dirichlet distribution).
Surface Projection: The point $v$ is projected onto the fitted local surface using a stabilized Newton-type iterative scheme to find $z^*$ such that $f(z^*) \approx 0$ .
Reconstruction: The point is mapped back to the original embedding space.

D. Evaluation Metrics

The paper distinguishes between Geometric Validity and Discriminative Utility:

Geometric Validity Criteria:
- Surface Consistency: Residual distance to the fitted surface.
- Neighborhood Consistency: Overlap of $k$ -nearest neighbors with original points.
- Distribution Consistency: Mahalanobis distance to the empirical cloud.
- Hessian-based Shape Consistency: Stability of second-order shape descriptors (curvature) after adding synthetic points.
- Coefficient Consistency: Stability of the fitted model's algebraic coefficients.
Discriminative Utility: Performance in a downstream few-shot classification task (Logistic Regression) with a context-held-out split (testing on unseen slot combinations).

3. Key Contributions

CoPaGE-300K Dataset: A novel, controlled template-based dataset with slot-level annotations and precomputed embeddings, designed specifically for studying local manifold structures.
Explicit Local Modeling: A framework for approximating local semantic patches using low-degree (quadratic/cubic) fitted surfaces in reduced PCA space, moving beyond simple linear assumptions.
Surface-Based Generation: A method for generating synthetic latent points that are explicitly aligned with the fitted local geometry, rather than just interpolating linearly.
Geometric Validity vs. Discriminative Utility: The introduction of a rigorous evaluation framework that separates the geometric correctness of synthetic points from their ability to improve classifier performance.
Geometric Validity Criteria: A suite of metrics (including Hessian-based shape consistency) to evaluate the stability of the local manifold structure when augmented with synthetic data.

4. Experimental Results

RQ1: Do controlled classes form nonlinear manifolds?

Yes. Nonlinear models (Quadric and Cubic) significantly outperform affine models in approximating the local embedding clouds.
Evidence: Quadric models reduced RMSE and MAE by orders of magnitude compared to affine models across all regimes. Cubic models offered further improvements in complex regimes (C5).
Visualization: PCA projections show clear curvilinear structures that linear models fail to capture.

RQ2: Can fitted surfaces generate geometrically valid points?

Yes, but with trade-offs.
Fitted-Geometry Fidelity: The surface-based method achieved near-zero surface residuals and the highest stability in Hessian-based shape and model coefficients. It best preserved the analytic structure of the local patch.
Empirical-Density Fidelity: Conversely, local random perturbation outperformed surface-based generation in preserving the empirical density and nearest-neighbor structure.
Conclusion: Surface-based generation is superior for maintaining the shape of the manifold, while perturbation is better for maintaining the density of the cloud.

RQ3: Does geometric validity improve downstream classification?

No. This is a critical finding.
Result: In few-shot, context-held-out classification tasks, adding synthetic points (whether linear, perturbed, or surface-based) did not systematically improve accuracy or Macro-F1 scores compared to the baseline (no synthetic points).
Interpretation: Geometric validity (being on the manifold) does not guarantee discriminative utility. If the embedding space already contains strong invariances, synthetic points on the manifold may be redundant for the classifier and fail to refine the decision boundary.

5. Significance and Implications

Reframing Data Augmentation: The paper argues that geometric validity and discriminative utility are distinct properties. A point can be geometrically perfect but useless for classification. This challenges the assumption that "better geometry" automatically equals "better performance."
Offline Diagnostic Tool: The proposed method is best viewed as an offline analysis tool for probing representation spaces, understanding local semantic neighborhoods, and detecting anomalies, rather than a universal "plug-and-play" classifier booster.
Future Applications: The ability to model local geometry explicitly opens doors for:
- Out-of-Distribution (OOD) detection.
- Adversarial filtering.
- Constructing hard positive examples for contrastive learning.
- Hallucination analysis and uncertainty estimation.
Methodological Rigor: By using controlled templates, the study isolates the geometric effects of semantic variation, providing a clearer picture of how embedding models encode meaning than is possible with noisy natural corpora.

In summary, the paper establishes that controlled semantic variations induce measurable, nonlinear local geometries in sentence embedding spaces that can be explicitly modeled. While surface-based generation successfully preserves this geometry, it does not automatically translate to improved downstream classification, highlighting the need for discriminative filtering when using synthetic data.