Text-as-Signal: Quantitative Semantic Scoring with Embeddings, Logprobs, and Noise Reduction

This paper introduces a practical "Text-as-Signal" pipeline that transforms text corpora into quantitative semantic signals by combining full-document embeddings, logprob-based scoring against a configurable dictionary, and noise-reduced dimensionality reduction, demonstrated through an analysis of 11,922 Portuguese AI news articles using Qwen embeddings and UMAP.

The Gist

Imagine you have a massive library containing nearly 12,000 news articles about Artificial Intelligence. Right now, this library is a chaotic mess. The books are piled up randomly, some are torn, some are just scribbles, and it's impossible to tell at a glance what the library is actually about or how the different books relate to one another.

This paper presents a clever, automated system to turn that messy pile of text into a clear, navigable map. The authors call this process "Text-as-Signal." Instead of just reading the words, they treat the text like a radio signal that can be measured, tuned, and plotted on a graph.

Here is how their system works, broken down into simple steps using everyday analogies:

1. The "DNA Scan" (Embeddings)

First, the system takes every single news article and gives it a unique "DNA scan." In technical terms, this is called an embedding.

• The Analogy: Imagine taking a photo of every book in the library and turning that photo into a long list of numbers (a code). This code captures the essence of the article. Two articles about "AI safety" will have very similar codes, while an article about "AI profits" will have a very different code.
• The Result: Suddenly, every article has a specific coordinate in a giant, invisible 4,000-dimensional space.

2. The "Shrink Ray" (UMAP Projection)

That 4,000-dimensional space is too complex for humans to see. So, the system uses a "shrink ray" (a technique called UMAP) to squish that giant space down into a flat, 2D map that we can actually look at.

• The Analogy: Think of it like taking a 3D globe and flattening it into a 2D map of the world. You lose a little bit of detail, but now you can see the continents and oceans clearly.
• The Result: You get a "topographic map" of the news. Clusters of articles that talk about similar things group together, forming "islands" or "continents" of topics.

3. The "Six-Point Compass" (Logprob Scoring)

A map is useless without a compass. The authors created a configurable dictionary with six specific directions (dimensions) to measure every article against.

• The Analogy: Imagine a compass that doesn't just point North, but has six dials:
  1. Opportunity vs. Risk: Is the article optimistic or scary?
  2. Regulatory Pressure: Is it talking about rules or freedom?
  3. Economic Momentum: Is it about niche ideas or big money?
  4. Ethics vs. Utility: Is it about human values or just efficiency?
  5. Geopolitical Scope: Is it local or global?
  6. Urgency: Is it a calm analysis or a breaking news alarm?
• The Magic: Instead of asking a human to read and label every article, the AI looks at the text and calculates a score from 0 to 1 for each dial. It's like the AI is whispering, "This article is 80% 'Opportunity' and 20% 'Risk'."

4. The "Noise Filter" (Anomaly Detection)

Not all data is good data. Some articles are weird outliers, typos, or just irrelevant noise that messes up the map. The system runs a three-step "quality control" check to clean the map.

• The Analogy: Imagine a crowded party.
  • Step 1 (Global Outliers): The system kicks out the people standing outside the building entirely (articles that don't fit the theme at all).
  • Step 2 (Local Mavericks): Inside the party, it asks, "Is anyone standing in the corner screaming alone while everyone else is chatting?" If so, it marks them as weird outliers.
  • Step 3 (Disconnected Islands): It checks if there are tiny, isolated groups of people who are cut off from the main crowd and removes them.
• The Result: The final map is "clean." It only shows the stable, coherent groups of articles, making the patterns much easier to see.

What Did They Find?

When they applied this to 11,922 Portuguese news articles about AI, they discovered some interesting things:

• The "Sweet Spot": Most articles were "Opportunity" focused and "Analytical" rather than "Crisis" focused.
• The Map Works: The "Opportunity" articles clustered in one corner of the map, while "Risk" articles clustered in another. The six compass dials perfectly matched the geography of the map.
• No Human Needed: They didn't need a team of humans to read and tag every article. The system did it automatically, turning text into data that a computer can use for monitoring and decision-making.

Why Does This Matter?

Usually, when companies have thousands of documents, they are stuck. They can't easily search for "all articles that are high-risk but low-urgency" without reading them all.

This paper shows a way to turn text into a dashboard.

• For a CEO: They can look at the map and instantly see, "Oh, our news coverage is too focused on 'Ethics' and not enough on 'Economic Growth'."
• For a Monitor: They can set an alarm. If the "Risk" dial suddenly spikes for a whole cluster of articles, the system alerts them immediately.

In short: The authors built a machine that reads a messy pile of news, cleans it up, gives every article a scorecard on six different topics, and draws a map showing exactly where everything fits. It turns "reading" into "measuring."

Technical Summary

1. Problem Statement

Dense embeddings are effective for representing text, but raw vector spaces are often difficult to utilize in operational AI engineering settings. Traditional approaches rely on indirect interpretation of latent spaces or generative labeling (prompting models for text), which can be unstable, discrete, and unscalable.
The core challenge addressed is how to transform unstructured text corpora into quantitative, continuous semantic signals that can be directly used for downstream tasks such as aggregation, monitoring, regression, and threshold-based routing without requiring human analysts to interpret latent geometry.

2. Methodology

The authors propose a four-stage pipeline to convert a text corpus into a stable, noise-reduced semantic manifold with continuous scoring.

A. Document Embedding

• Unit of Analysis: Full news articles (no chunking).
• Model: Qwen2.5 8B Instruct model.
• Output: 4096-dimensional vectors stored in PostgreSQL (pgvector).
• Rationale: Selected for strong open-weight performance on text-embedding benchmarks (MTEB) and structure-preserving capabilities.

B. Structural Reduction & Partitioning

• Dimensionality Reduction: The 4096D space is projected via UMAP into:
  • A 5D latent space for structural analysis (aligned with the corpus's estimated intrinsic dimensionality of ~4.11).
  • A 2D space for visualization.
• Clustering: K-Means ($K=15$) is applied to the 5D manifold to create initial structural partitions.
• Exploratory Tool: HDBSCAN is used only as a density diagnostic, not for final partitioning, as it was found too restrictive (classifying ~45% of data as noise).

C. Logprob-Based Semantic Scoring (The "Identity Layer")

Instead of generating text labels, the pipeline treats the LLM's output distribution as a signal evaluator.

• Mechanism: For each article, the model compares the final hidden state against target label vectors (poles) via a softmax layer to generate logprob-based scores.
• Scoring Formula: A normalized score $s_{i,m} \in [0, 1]$ is calculated for each dimension $m$:
  $$s_{i,m} = \frac{\exp(\lambda^+_{i,m})}{\exp(\lambda^-_{i,m}) + \exp(\lambda^+_{i,m})}$$
  Where $\lambda$ represents log-scores for the lower ($-$) and upper ($+$) poles.
• The 6 Semantic Dimensions:
  1. Opportunity vs. Risk
  2. Regulatory Pressure (Deregulation vs. Compliance)
  3. Economic Momentum (Niche vs. Commercial)
  4. Ethics vs. Utility (Human-centric vs. Efficiency)
  5. Geopolitical Scope (Local/EU vs. Global)
  6. Urgency (Analytical vs. Breaking/Alarmist)

D. Three-Stage Noise Reduction (Anomaly Detection)

To stabilize the topology, a three-step filtering process removes outliers:

1. Global Outlier Filter: Removes points far from the global centroid of the HDBSCAN-defined "continental core" (threshold: $\mu + 1.2\sigma$).
2. Local Maverick Filter: Removes points that are anomalous relative to their specific K-Means cluster centroid (threshold: $\mu + 1.8\sigma$).
3. Structural Connectivity Filter: Uses a graph-based approach (inspired by SCAN) to remove structurally disconnected islands. Only articles in the largest connected component are retained.

3. Case Study & Results

• Dataset: 11,922 Portuguese news articles (2022–2024) regarding Artificial Intelligence.
• Noise Reduction Impact:
  • Initial K-Means yielded 15 regions.
  • The filtering process removed 2,565 articles (21.5%), including 1,282 global outliers and 944 local mavericks.
  • The final stable map consists of 13 structural regions.
• Key Findings:
  • Semantic Centrality: The corpus showed a strong peak in centrality scores (0.6–0.75) for "Artificial Intelligence," validating the keyword filtering. Crucially, the removed outliers were not just low-centrality noise but topologically unstable documents distributed across the spectrum.
  • Aggregate Profiling: The corpus was characterized as predominantly Opportunity-oriented (89% in Q3/Q4), Low Supervision (49% in Q2), Commercial Growth (67% in Q3), and Balanced Ethics (94% in Q2). Extreme risk and crisis narratives were rare.
  • Spatial-Semantic Alignment: When logprob scores were projected onto the 2D manifold, opposite poles (e.g., Risk vs. Opportunity) occupied distinct, localized spatial regions, proving that the geometric structure aligns with semantic meaning.

4. Key Contributions

1. Operational Text-as-Signal Pipeline: A concrete workflow that transforms text into continuous, bounded variables suitable for AI engineering tasks (monitoring, regression, routing) rather than just visual exploration.
2. Logprob-Based Evaluation: Demonstrates that reading the model's output distribution (logprobs) provides a more stable, continuous, and calibrated signal than generative text labeling.
3. Configurable Identity Layer: The semantic dictionary is not fixed; it can be reconfigured for different analytical streams (e.g., specific companies or events) without changing the underlying pipeline.
4. Hybrid Noise Reduction: A novel combination of geometric distance (global/local) and topological connectivity (graph-based) to create a robust, interpretable semantic map.

5. Significance

• AI Engineering Utility: Moves semantic analysis from a "visual exercise" to a functional component of data pipelines, enabling automated monitoring and decision support.
• Scalability: By avoiding generative text extraction, the method is computationally efficient and scalable to large corpora.
• Interpretability: It bridges the gap between dense vector geometry and human-readable semantic concepts, allowing analysts to "read" the map both geometrically and semantically.
• Adaptability: The framework allows for the redefinition of operative dimensions based on specific domain requirements, making it a flexible tool for diverse use cases beyond the demonstrated AI news analysis.

Limitations & Future Work

• Limitations: The design choices (e.g., $K=15$, anomaly thresholds) were validated operationally rather than through exhaustive benchmarking. The method relies on prompt anchoring and specific model behaviors.
• Future Directions: Extending the pipeline to temporal analysis (tracking drift over time), using identity profiles as explanatory variables in forecasting models, and analyzing the "removed" noise for emergent weak signals rather than discarding them.