Technical Summary: The Model Knows Your Project, Not You: Measuring Recognition in LLMs with NameRank

1. Problem Statement

Frontier Large Language Models (LLMs) increasingly serve as the primary interface for human curiosity regarding people and their work. Before any retrieval-augmented generation (RAG) occurs, a model's parametric memory determines whether a user receives a coherent summary or a hallucination/unknown response. However, existing metrics fail to measure this "corpus presence" effectively.

Bibliometric indicators (e.g., citation counts, h-index) explain only about one-third of the variance in whether a model recognizes a researcher. The remaining variance is driven by mechanisms such as name uniqueness, attachment to named artifacts, and subfield ecosystem density, but these have not been quantified on a unified scale. Furthermore, traditional benchmarks often conflate recognition with reach or fail to distinguish between genuine knowledge and fluent hallucinations. The paper posits that recognition is not a function of credentials or titles, but of whether a name has propagated into the training weights of frontier models as a distinct, indexable entity.

2. Methodology: NameRank

The authors introduce NameRank, a cross-domain recognition instrument designed to measure the probability that a randomly drawn frontier model genuinely knows a specific entity.

2.1 The Panel and Probes

• Panel: 36 frontier and near-frontier models (spanning Western and Chinese vendors, including reasoning-enabled variants) accessed via OpenRouter and direct APIs.
• Entities: 4,685 entities across 54 cohorts, including credentialed individuals (Olympiad medalists, Fellows), working academics, industry figures, open-source maintainers, and named artifacts (tools, methods, datasets).
• Probe Design: A fixed, open-ended prompt is used for each entity: "Tell me what you know about [name], who [context]. If you do not recognize this entity, respond with 'unknown'. Limit your response to about 150 words."
  • The context provides role, affiliation, and field to disambiguate names without revealing specific contributions or the "gold" answer.
  • Open-ended probing avoids the failure mode of closed-factoid probes, where a model knowing Fact X but not Fact Y would score zero.

2.2 The Recognition Verdict

An independent LLM judge (Gemini 3 Flash Preview) evaluates each response against a curated Gold Answer.

• Binary Verdict: The judge returns 1 (recognized) only if the response states at least one specific, non-guessable, and correct fact about the entity that is:
  1. Verified against the gold answer or the judge's reliable knowledge.
  2. Not derivable from the probe context.
  3. Not a restatement of the context or a generic biography.
• Penalties: Fluent hallucinations, context echoes, lucky guesses, and refusals score 0.
• Scoring: The NameRank score for an entity is the unweighted panel recognition rate (the fraction of the 36 models that return 1).

2.3 Calibration and Validation

• Synthetic-Null Floor: Fictional entities (plausible but non-existent researchers, founders, etc.) are probed to establish a baseline. These entities score near zero (≤0.02), confirming that non-zero scores represent genuine recognition rather than guessing artifacts.
• Variance Decomposition: The entity factor explains 56% of the variance in verdicts, while the model factor explains only 9%, indicating the score measures the entity's presence in the corpus, not the specific model's quirks.
• Robustness Checks: The study validates results against wording variations, model vintages, judge families, and language (English vs. Chinese prompts), showing that rankings remain stable even if absolute levels shift.

3. Key Results

3.1 The "Artifact-Over-Creator" Mechanism

The central thesis is that recognition is paid to named, indexable artifacts, not to credentials, titles, or reach.

• Credentials vs. Artifacts: "Olympic-style" credentials (IMO, IOI, Putnam, Rhodes) score at or below the baseline for working researchers (e.g., IMO Gold: ~0.12 vs. Working Researcher: ~0.40). These credentials propagate names only if they are attached to subsequent named production (papers, tools).
• Named Artifacts: Named methods (e.g., ReAct, DPO), open-source projects (e.g., LangChain, Tianshou), and companies often out-rank their creators. For example, the tool Perplexity (0.89) scores higher than its founder Aravind Srinivas (0.75), who in turn scores higher than co-founder Johnny Ho (0.28), despite Ho being a three-time IOI gold medalist.
• Inversion: The artifact's name travels further in the corpus than the creator's name. This is an observational property of attribution asymmetry in indexable text, not something a prompt intervention can manufacture.

3.2 Predictors of Recognition

• Bibliometrics: No bibliometric (h-index, raw citations) predicts recognition well. Even the best correlates explain only ~22% of the variance.
• Corpus Density: Recognition is strongly correlated with the "corpus density" of the institution and country. Researchers at high-density institutions (MIT, Berkeley) score higher than peers at matched citation levels at lower-density institutions. Similarly, US-based researchers score significantly higher than Chinese or Indian counterparts, driven by the density of the English-language ecosystem (newsletters, podcasts, tutorials) rather than research output alone.
• News Events: For news events, recognition tracks peak salience (how loud the story was) rather than persistence (how long it lasted). High peak attention generates more name-bearing documents, which drives recognition.

3.3 Model Self-Report vs. Reality

When models are asked to self-report their recognition of entities, their rankings correlate with the shared corpus prior (general fame) rather than their own verified knowledge. Models recite consensus fame rather than introspecting their specific weights.

4. Contributions

1. A Cross-Domain Instrument: NameRank places researchers, founders, OSS authors, and named artifacts on a single [0, 1] recognition axis, calibrated by synthetic-null floors.
2. Career-Arc Analysis: It demonstrates that credentials only propagate names if they carry named artifacts. Best-paper awards and named-method authorship score above the researcher baseline; bare medals do not.
3. The Named-Artifact Mechanism: It empirically establishes that artifacts often out-rank their makers and that this inversion is an observational property of corpus attribution, not a prompt artifact.
4. Predictive Limits: It shows that bibliometrics are poor predictors of LLM recognition and that corpus density (institution/country) and peak salience (for events) are stronger drivers.
5. Validation Battery: It provides a rigorous validation suite (including self-report probes and cross-language checks) and releases all data, probes, and code.

5. Significance and Claims

The paper claims that NameRank measures the residual of recognition that bibliometrics miss: the extent to which an entity's name has been internalized by frontier models as a distinct, indexable fact.

• Measurement Shift: As human curiosity is routed through models, the "presence" of a name in model weights becomes a substantive measurement problem. NameRank quantifies this presence.
• Structural Inequality: The instrument reveals structural inequalities in name propagation. Identical research output propagates differently depending on the English-corpus density of the researcher's ecosystem (e.g., US vs. China/India).
• Practical Implication: For individuals seeking recognition in the LLM era, the strategy is not merely to accumulate credentials or citations, but to produce distinctively named, indexable artifacts (tools, methods, papers) that accumulate in the corpus. Recognition is a cumulative ledger of name-bearing text, not a gauge of current attention or raw intellectual selectivity.
• Limitations: The paper modestly notes that absolute levels are wording-conditional and that the metric inherits demographic biases present in the training corpus and the judge. It measures what models have absorbed, not necessarily the "truth" or "merit" of the entity.

In summary, the paper argues that in the LLM-mediated regime, the model knows your project, not you. Recognition is a property of the named artifact, not the credential.