💬 NLP

ENTLORE: A Graph-Grounded Benchmark for Latent Organizational Reasoning in Enterprise Question Answering

This paper introduces ENTLORE, a graph-grounded benchmark that evaluates enterprise question answering systems on their ability to perform latent organizational reasoning by reconstructing audited enterprise worlds from routine documents to test the recovery of implicit relations beyond simple fact retrieval.

Original authors: Akrin Zheng, Alexander Wu, Alaia Liu

Published 2026-08-12

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Akrin Zheng, Alexander Wu, Alaia Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery inside a giant, chaotic library. This isn't just any library; it's the "brain" of a massive company, filled with millions of documents like emails, project reports, meeting notes, and spreadsheets. In the world of Artificial Intelligence (AI), there is a popular game called "Question Answering." Usually, the AI is given a question and a stack of documents, and its job is to find the exact sentence that holds the answer. Think of it like a game of "I Spy" where the answer is hidden in plain sight, waiting to be picked up.

But real life in a company is messier than a game. Often, the answer to a question isn't written down in a single sentence. Instead, the truth is hidden in the relationships between different documents. Maybe one email says "Alice worked on Module X," and another report says "Module X is part of Project Y," but no single document ever says "Alice worked on Project Y." To find the answer, the AI has to connect the dots itself, using the invisible rules of how the company is organized. This paper, written by researchers at ScitiX.ai, is all about teaching computers how to do that kind of detective work, rather than just being good at finding words.

The Problem: The "Hidden Truth" Trap

Most AI benchmarks (tests for AI) are like a treasure hunt where the map is already drawn. The test creators write a question, then they make sure the answer is explicitly written in the documents they give the AI. It's like asking, "What color is the red ball?" and handing the AI a picture of a red ball. The AI just has to find the word "red."

The authors of this paper argue that this is too easy and doesn't reflect how real companies work. In the real world, the "red ball" might be mentioned in one file, and the fact that it's a "ball" might be in another, but the connection between them is never stated. The AI has to figure out that "Module X" belongs to "Project Y" based on how the company is structured, even if no document ever explicitly links them. The researchers call this missing skill Latent Organizational Reasoning. It's the ability to infer the invisible rules of an organization from the messy by-products of daily work.

The Solution: ENTLORE, the "Truth Graph"

To test if AI can actually do this, the team built a new benchmark called ENTLORE. Imagine they took a real, private company's entire digital history—thousands of documents, organizational charts, and operational logs—and built a giant, secret "Truth Graph" behind the scenes. This graph is like a master blueprint that knows exactly how every person, project, and module connects, based on strict, audited rules.

Then, they created a "released world" for the AI to play in. This released world contains the same documents, but all the names are changed to protect privacy (like turning "Alice" into "Employee #42" and "Project Alpha" into "Project Beta"). Crucially, the AI cannot see the secret Truth Graph. It only sees the messy, anonymized documents.

The team then asked 907 questions. They broke these questions down into three levels of difficulty:

Level 1 (The Lookup): The answer is right there in one document. (e.g., "Who is the manager of Project Beta?")
Level 2 (The Composition): The answer requires reading two documents and combining facts. (e.g., "Who manages the team that built Module X?")
Level 3 (The Latent Reasoning): The answer requires connecting dots that aren't explicitly linked. (e.g., "Who is responsible for the overall success of Project Beta?"—even though no document says "Employee #42 is responsible for Project Beta," the Truth Graph knows it because they managed the module inside it.)

The Findings: AI is Good at Reading, Bad at Connecting

The researchers tested 8 different AI models using various methods, from simple keyword search to complex "agent" systems that can browse documents like a human. Here is what they found:

1. The "Hidden Truth" is Hard to Find
When the answer was explicitly written in the documents (Levels 1 and 2), the AI models did reasonably well. But when they hit Level 3 (Latent Reasoning), the performance dropped dramatically. Even the best models only got about 36.2% of these hard questions right.

2. Structure Matters More Than Speed
The researchers tried different ways to help the AI find information.

Simple Search (BM25): Just looking for matching words. This was surprisingly good.
Dense Search (Flat RAG): Using advanced math to find similar ideas. This actually did worse than simple search.
Graph-Based Systems (GraphRAG): These systems try to build a map of connections between documents before answering. This turned out to be the strongest approach, beating the others on average. It suggests that AI needs to understand the structure of the company, not just the words.

3. Even with the "Gold Documents," AI Fails
The most shocking discovery came when the researchers gave the AI the "Gold Documents"—the exact files needed to answer the question, skipping the search part entirely.

For easy questions (Level 1), the AI got almost everything right.
For medium questions (Level 2), it still did well.
But for the hard, latent questions (Level 3), 30.4% of the answers were still wrong, even though the AI had the perfect documents in front of it!

This means the problem isn't just that the AI can't find the right file; it's that it can't reason through the invisible organizational rules even when it has all the evidence. It's like giving a detective the crime scene photos but still having them fail to realize that the butler and the gardener are actually the same person because the photos don't explicitly say so.

Why This Matters

The paper concludes that we can't just build better search engines for companies. We need AI that can understand the "soul" of an organization—the unspoken rules, the project hierarchies, and the hidden connections. ENTLORE proves that current AI is still struggling with this. It's great at reading what is written, but it's still learning how to think about what is implied.

The researchers made their data and code public so other scientists can try to solve this puzzle. Until AI can master this "latent organizational reasoning," it will always be a bit like a new employee who knows how to read the handbook but doesn't yet understand how the company actually works.

Technical Summary: ENTLORE – A Graph-Grounded Benchmark for Latent Organizational Reasoning

Problem Statement

Enterprise Question Answering (QA) is traditionally framed as a retrieval task where systems identify internal documents to generate grounded answers. However, routine enterprise records are often work-by-products rather than self-contained descriptions designed for future querying. Consequently, many critical organizational relations—such as project scopes, alias correspondences, or workstream responsibilities—remain implicit across heterogeneous sources.

Existing benchmarks (e.g., TechQA, EKRAG, MultiDoc2Dial) have improved multi-source evaluation but typically materialize evidence along a predefined reasoning path. In these setups, every required step is explicitly stated in the corpus, testing a system's ability to retrieve and compose stated facts rather than recover a target relation that is absent from the text. This paper identifies a gap in current evaluation: the inability to assess latent organizational reasoning, defined as the capability to recover and reason over organizational relations that are not explicitly stated in any released document but are derivable from the underlying organizational structure.

Methodology: The ENTLORE Framework

The authors introduce ENTLORE, a graph-grounded benchmark construction framework designed to evaluate this latent reasoning capability without exposing the ground truth to the evaluated systems. The framework operates in four stages:

Reconstruction of the Audited Truth Graph ( $G_{truth}$ ):
- The system reconstructs a raw graph ( $G_{raw}$ ) from routine documents, authoritative organizational tables, and operational traces.
- A versioned library of organizational conventions (private, typed inference rules) is applied to certify derived relations. For example, a rule might infer that a person works on a project if they work on a module contained within that project.
- The resulting truth graph separates native facts, metadata, heuristic associations, and certified derived relations.
Privacy-Preserving Corpus Projection:
- A shared anonymization map ( $\phi$ ) projects the private world into a released document corpus ( $D_{release}$ ).
- This process consistently renames entities and shifts dates while preserving relative organization, genre, and information density.
- Crucially, $D_{release}$ contains no explicit statements of the target relations. The private organizational records, certification rules, and the truth graph remain hidden from the evaluated systems.
Question Construction and Capability Decomposition:
- Questions are generated via typed graph operators that sample from $G_{truth}$ , execute queries, and record proof dependencies.
- The benchmark categorizes 907 questions into three levels of difficulty based on evidence availability:
  - L1 (Stated Lookup): The answer is explicitly stated in a single released document.
  - L2 (Stated Composition): The answer requires composing facts explicitly stated across multiple released sources.
  - L3 (Latent Organizational Reasoning): The target relation appears in no released source. It exists only in $G_{truth}$ and must be recovered by inferring implicit organizational structures from distributed observations.
Verification:
- Every question is verified to ensure the graph program returns a unique, complete answer with certified derivations.
- For L3 items, a release-wide scan confirms the target relation is absent from all released entity aliases.

Key Contributions

Formulation of Latent Organizational Reasoning: The paper defines a new capability where systems must combine heterogeneous grounded facts to recover organization-specific relations that are not explicitly stated.
Audited Truth Graph Construction: A framework for building a benchmark from real enterprise documents and authoritative tables, separating native facts from certified derived relations.
The ENTLORE Benchmark: A dataset containing 2,341 documents and 907 questions (469 L1, 204 L2, 234 L3) evaluated across 56 model and access configurations.
Evaluation of Access Paradigms: A comprehensive matrix comparing closed-book, lexical retrieval (BM25), flat dense retrieval (RAG), agentic pipelines, and graph-based approaches (GraphRAG, LLM Wiki) against the latent reasoning challenge.

Experimental Results

The evaluation spans eight models (including GPT-5.4, Claude-Sonnet, Qwen, DeepSeek) and seven access conditions.

Performance on Stated Evidence (L1/L2): Systems perform reasonably well on explicit lookup and composition. GraphRAG, BM25, and an LLM-compiled Wiki form the top tier, with accuracies ranging from ~50% to 63%. Flat RAG and agentic retrieval generally underperform these structured or lexical baselines.
Performance on Latent Reasoning (L3): Performance drops significantly across all configurations.
- GraphRAG achieves the highest average L3 accuracy at 36.2%, followed by BM25 (34.0%) and the LLM Wiki (27.7%).
- Flat RAG (18.9%) and Agentic retrieval (13.5%) perform poorly, suggesting that iterative interaction without structural induction does not help recover latent relations.
- The performance gap is highly heterogeneous; GraphRAG significantly outperforms BM25 on "Department attribution" tasks but shows less advantage on "Containment attribution."
The Gold Document Residual: Even when systems are provided with the exact "gold documents" containing all necessary evidence (via an Oracle agent, $\Omega$ ), 30.4% of L3 questions remain unanswered. This is compared to 12.6% for L1 and 6.2% for L2. This residual indicates that the failure is not merely due to document retrieval (recall) but stems from the inability to perform the necessary organizational derivation even when all premises are present.
Release Fidelity: An analysis of 176 paired items between the private and released worlds shows that while the coarse ranking of oracle configurations is preserved, the fine-grained ordering of deployable systems shifts (Spearman's $\rho = 0.71$ , $p=0.06$ ). The authors conclude the release is valid for group-level comparisons but tentative for fine-grained ranking, particularly for lexical methods sensitive to anonymization.

Significance and Claims

The paper claims that Enterprise QA depends not only on document recall but critically on whether implicit organizational relations become usable.

Beyond Retrieval: The results demonstrate that supplying gold documents does not solve the latent reasoning problem. The "latent-relation challenge" persists even when retrieval errors are eliminated.
Structure as a First-Class Object: The findings suggest that enterprise QA systems must treat the organization behind the documents as a first-class inference object, recovering typed relations across sources and preserving provenance through multi-step reasoning.
Benchmark Utility: ENTLORE provides a testbed for measuring progress in organizational reasoning without reducing the task to simple document recall. By keeping the benchmark truth private but its premises auditable, it allows for the evaluation of systems that can reason over implicit structures.

The authors remain modest regarding the release fidelity, noting that while the benchmark preserves comparative behavior for broad groups, the specific ranking of deployable systems may vary due to the sensitivity of lexical retrieval to anonymized text surfaces. They emphasize that the benchmark is a step toward evaluating systems capable of handling the "work-by-product" nature of real enterprise data.