OrgForge: A Multi-Agent Simulation Framework for Verifiable Synthetic Corporate Corpora

OrgForge is an open-source multi-agent simulation framework that generates verifiable, temporally consistent synthetic corporate corpora by decoupling a deterministic event-ground-truth engine from LLM-based surface prose generation to eliminate hallucinations and timeline inconsistencies for robust RAG pipeline evaluation.

The Gist

Imagine you are trying to teach a robot how to be a helpful office assistant. You want to test if it can find the right information in a massive pile of company emails, chat messages, and project tickets when something goes wrong.

The problem? Real company data is messy, private, and often contradictory. If you make up fake data using a standard AI, the AI might accidentally lie to itself—saying a server crashed at 3 AM in one email but at 9 AM in a project ticket. If the robot learns from that, it will fail in the real world.

OrgForge is a new tool created by researcher Jeffrey Flynt to solve this. Think of it as a "Digital Movie Studio" for corporate chaos, but with a very strict director who never lets the actors improvise the plot.

Here is how it works, broken down into simple concepts:

1. The Strict Director vs. The Actors

In most AI systems, the AI writes the story and decides the facts. In OrgForge, they are separated:

• The Engine (The Director): A rigid, math-based computer program runs the "physics" of the company. It decides exactly when a server crashes, who is on call, how stressed an employee is, and what the exact time is. It keeps a perfect, unchangeable log of every single fact.
• The LLMs (The Actors): Large Language Models are hired only to write the dialogue. They take the Director's facts ("The server crashed at 3 AM, and Bob is stressed") and write a realistic Slack message or email. They cannot change the facts; they can only dress them up in words.

The Analogy: Imagine a courtroom. The Judge (the Engine) dictates the timeline and the evidence. The Lawyers (the LLMs) argue the case and write the speeches. The lawyers can be dramatic, but they cannot lie about what the Judge has already recorded in the official log.

2. The "Social Stress" Game

OrgForge doesn't just make random noise; it simulates how real people react to pressure.

• The Stress Battery: Every employee has a hidden "stress battery." When a crisis hits, the battery drains.
• The Ripple Effect: If a key person (like a team lead) gets overwhelmed, their stress "bleeds" onto their teammates, just like a heavy backpack makes everyone in a hiking group slower.
• The Friendship Map: The system tracks who talks to whom. If two people work together on a crisis, their "friendship bond" gets stronger. If they ignore each other for a week, the bond weakens. This happens automatically, without the AI guessing.

3. The "Time Travel" Fix

One of the biggest problems with fake data is time travel. A fake email might say, "I saw the error," but the error log says the error happened after the email was sent.
OrgForge uses a "Personal Clock" for every employee.

• If Bob is fixing a bug, his clock ticks forward.
• If Alice replies to Bob, her clock must be ahead of Bob's.
• The system forces the timeline to flow forward logically, ensuring that cause always comes before effect. No time travel allowed!

4. The "Lost Email" Test

Real offices are full of noise. Sometimes an email gets lost, or a manager ignores a complaint.
OrgForge intentionally simulates this. It might generate a customer complaint email that never gets answered.

• Why? To test the robot. If the robot is asked, "Did we fix the customer's complaint?" and the answer is "No, because the email was dropped," the robot needs to realize that the absence of a reply is a valid answer. Most AI systems get confused when the answer is "nothing happened."

5. The Final Exam

Once the simulation runs for 30 days, OrgForge produces a massive dataset of fake but perfectly consistent company documents. It then automatically generates a test:

• Question: "Who was the first person to know about the database crash on Tuesday?"
• The Answer: The system knows the exact answer because it wrote the log.
• The Score: It checks if the robot found the right email, the right ticket, and the right timeline.

Why This Matters

Before OrgForge, testing AI on corporate data was like testing a driver on a track with invisible walls. You didn't know if they crashed because they were bad drivers or because the track was broken.

OrgForge builds a track with perfectly marked walls. It allows researchers to see exactly where an AI fails:

• Did it miss the email?
• Did it get the timeline wrong?
• Did it hallucinate a person who doesn't exist?

By separating the "facts" from the "words," OrgForge gives us a way to build and test AI assistants that are actually ready for the messy, complex reality of the modern office.

Technical Summary

1. Problem Statement

The evaluation of Retrieval-Augmented Generation (RAG) systems in enterprise settings faces a critical bottleneck: the lack of high-quality, ground-truth-annotated datasets that reflect the complexity of real-world corporate knowledge.

• Limitations of Real Data: Existing real-world datasets (e.g., the Enron corpus) suffer from legal ambiguities, demographic skews, and a lack of structured ground truth. Researchers cannot verify if a RAG system retrieved the "correct" fact because the actual sequence of events is not explicitly recorded.
• Limitations of Synthetic Data: Purely Large Language Model (LLM)-generated corpora solve legal issues but introduce factual inconsistency. LLMs often hallucinate contradictions across documents (e.g., a Slack message stating an incident started at 3 AM while a JIRA ticket says 9 AM). Without an external truth source, these inconsistencies silently corrupt RAG evaluations.

The Core Challenge: How to generate synthetic corporate data that is realistic, cross-artifact (spanning Slack, JIRA, Email, etc.), temporally structured, and mathematically guaranteed to be internally consistent with a known ground truth.

2. Methodology: The OrgForge Architecture

OrgForge is an open-source, multi-agent simulation framework that enforces a strict "Physics-Cognition Boundary." It separates the deterministic simulation of facts from the probabilistic generation of prose.

A. Formal System Definition ($M = (S, P, V, E)$)

The system is defined as a tuple where:

• $S$ (State): A Pydantic model representing mutable variables (system health, team morale, stress levels, open tickets).
• $P$ (Planners): LLM-based agents that propose daily activities (e.g., "hold a meeting," "fix a bug"). They influence the narrative but cannot directly modify the state or write to the event log.
• $V$ (Validator): A deterministic function that screens LLM proposals against the current state $S$ and history $E$. It rejects hallucinations (e.g., non-existent employees, impossible events during a crisis) before execution.
• $E$ (Events): The SimEvent log. This is the single source of truth (ground truth bus). Every significant action emits a structured SimEvent to a persistent log. LLMs generate surface prose only based on context retrieved from $E$.

B. Key Subsystems

1. Graph Dynamics (Deterministic Organizational Behavior):

   • Stress Propagation: Uses Betweenness Centrality to calculate how stress spreads from "burnt-out" key players to their network. Stress is clamped and decays daily.
   • Edge-Weight Decay: Relationships (edges) decay over time without interaction but strengthen via collaboration (e.g., joint incident response, PR reviews).
   • Escalation Routing: Models incident escalation as a Shortest-Path problem (Dijkstra's algorithm) on an inverse-weight graph. Incidents flow through the strongest existing communication bonds.

2. Actor-Local Clock (sim_clock.py):

   • Solves the "timeline inconsistency" problem. Instead of sampling timestamps independently per document, every employee maintains an independent time cursor.
   • Timestamps are derived from cursor state, ensuring causal correctness (e.g., a response cannot be timestamped before the trigger).

3. Artifact Generation & Validation:

   • Generates interleaved artifacts: Slack threads, JIRA tickets, Confluence pages, Git PRs, emails, and server logs.
   • Proposal-Validation Loop: Ensures actors exist, events are plausible given system health (e.g., no celebrations during a P1 outage), and cooldowns are respected.

4. External & Causal Systems:

   • Causal Chain Tracking: Accumulates ordered artifact graphs per incident, linking server logs $\to$ alerts $\to$ tickets $\to$ postmortems.
   • Recurrence Detection: Uses Hybrid Reciprocal Rank Fusion (RRF) to detect repeated failure classes by fusing vector and text search over the SimEvent log.
   • Email Engine: Simulates inbound vendor alerts and customer complaints. Includes probabilistic drop simulation (15% of emails are dropped) to create verifiable "gaps" for retrieval evaluation.
   • Ambient Social Interruption: Introduces "benign noise" (watercooler chats) to consume agent capacity, creating realistic retrieval challenges where low-signal artifacts must be filtered.

3. Evaluation Pipeline

OrgForge includes a complete evaluation harness (eval_harness.py, scorer.py) that generates 8 types of questions with deterministic ground truth:

1. Retrieval: Finding the root cause artifact.
2. Causal: Identifying direct consequences of an event.
3. Temporal: Determining system state at a specific time.
4. Gap Detection: Identifying if an email was actioned or dropped.
5. Routing: Tracing who received a complaint first.
6. Planning: Identifying departmental themes.
7. Escalation: Mapping the escalation chain.
8. Knowledge Gap: Identifying undocumented domains.

Scoring: Uses a weighted formula ($0.80 \times$ Answer Correctness + $0.20 \times$ Evidence Retrieval Quality) to penalize lucky retrievals and reward correct reasoning.

4. Results

The authors ran a 22-business-day simulation with 43 personnel, generating 1,079 documents and 83 evaluation questions.

• Cost: Approximately $285.30 for ~504M input tokens and ~349M output tokens (using gpt-oss-120b-1).
• Baseline Performance:
  • BM25 (Keyword): Outperformed dense retrieval on most metrics (Overall MRR@10: 0.28 vs. 0.20). It excelled at Causal questions (MRR@10: 0.54) due to specific root-cause terminology in postmortems.
  • Dense Retrieval (Embeddings): Performed poorly on Temporal and Escalation questions (MRR@10: 0.00), indicating that current 1.5B parameter embeddings struggle with multi-hop reasoning and absence-of-evidence tasks.
  • Zero-Performance Areas: Both methods scored 0.00 on Planning and Escalation questions, highlighting a significant gap for future agentic systems.
• Key Finding: The corpus is internally consistent by construction. The "unconstrained" LLM generation (without the SimEvent bus) would likely show high contradiction rates, whereas OrgForge guarantees zero contradictions.

5. Key Contributions

1. Architectural Boundary: A formal separation of fact control (deterministic engine) from prose generation (LLM), preventing hallucination contamination.
2. Formal Graph Dynamics: Three mathematically specified mechanisms (Stress Propagation, Edge Decay, Dijkstra Escalation) that govern organizational behavior without LLM intervention.
3. Causal Timestamping: An actor-local clock system that guarantees causal consistency across all artifact types, eliminating timeline errors common in synthetic data.
4. Ground-Truth Noise: An ambient social interruption model that introduces realistic capacity noise and "dropped" communication gaps for robust retrieval testing.
5. Open-Source Benchmark: A complete pipeline (simulation, question generation, scoring, and HuggingFace export) with a reproducible, MIT-licensed implementation.

6. Significance

OrgForge addresses a fundamental gap in RAG research: the inability to evaluate systems on temporal reasoning, cross-artifact coherence, and ground-truth consistency. By providing a synthetic environment where the "truth" is mathematically guaranteed, it allows researchers to:

• Quantify the exact failure modes of RAG systems (e.g., hallucination vs. retrieval failure).
• Benchmark multi-hop reasoning and temporal logic in a controlled setting.
• Develop evaluation metrics for "absence of evidence" (e.g., detecting dropped emails).

The paper argues that OrgForge provides the foundation for a new class of enterprise RAG benchmarks that are not only realistic but verifiable, moving the field beyond static Wikipedia-style QA toward dynamic, multi-agent corporate simulation.