SpheriCity: Designing Trustworthy Conversational AI for Sustainability Decision Support

This paper introduces SpheriCity, a provenance-first conversational AI prototype designed to enhance trustworthy knowledge sensemaking and cross-document synthesis in sustainability decision-making, validated by expert feedback highlighting the critical role of transparent sourcing and workflow alignment in building user trust.

The Gist

The Big Problem: Too Many Books, Too Little Time

Imagine you are a city planner trying to figure out how to stop plastic waste. You have a massive library of 36 different reports, each one as thick as a phone book (about 65 to 100 pages). These reports are full of data, charts, and policies from different cities around the world.

To answer just one question—like "How did City A handle plastic bags compared to City B?"—you would have to manually flip through hundreds of pages, cross-reference numbers, and try to connect dots that are scattered across different documents. It's like trying to build a puzzle where the pieces are hidden inside 36 different boxes, and you have to find them all by hand. It's slow, tiring, and easy to miss important connections.

The Proposed Solution: A "Super Librarian" AI

The researchers built a tool called SpheriCity. Think of it as a super-smart, super-fast librarian who has read all 36 of those thick reports and can chat with you about them.

You can ask it questions in plain English, like "What are the best ways to reduce plastic litter in coastal cities?" The AI scans the reports, finds the answers, and writes a summary for you.

But here's the catch: In the past, AI chatbots were like confident but forgetful students. They might give you a smooth-sounding answer that sounded great but was made up (a "hallucination") or didn't tell you where they got the info. In sustainability, where decisions affect the environment and public policy, you can't trust an answer if you can't check the homework.

How SpheriCity is Different: The "Receipt" Approach

The researchers realized that for experts (the city planners and scientists), trust is more important than speed. So, they designed SpheriCity with three special features:

1. The "Receipt" (Provenance): Every time the AI makes a claim, it attaches a digital "receipt." It doesn't just say "City X did this"; it says "City X did this, according to Page 42 of the 2023 Report." This lets the expert instantly click and verify the source.
2. The "Bullet Point" Summary (Structure): Instead of writing a long, confusing essay, the AI organizes answers into clear bullet points (like a shopping list). This makes it easy to scan, compare, and find missing information quickly.
3. The "Scaffolding" (Templates): The system gives experts pre-made question templates. Instead of struggling to phrase a complex question, they can select a template like "Compare policies across cities," which helps the AI understand exactly what the expert needs.

The Experiment: Testing with Real Experts

The team didn't just guess if this worked; they tested it with six real sustainability experts.

They gave the experts 13 different questions (like comparing recycling in India vs. the US) and asked them to rate the AI's answers. They tested three different "brain" strategies for the AI:

• Vector Search: Looking for words that sound similar.
• Graph Search: Looking for how ideas are connected (like a map of relationships).
• Hybrid: A mix of both.

The experts rated the answers on things like: Is this relevant? Is it accurate? Is it neutral? Is it deep enough?

What They Found: What Experts Actually Care About

The results were surprising and very practical. Here is what the experts told them:

• Trust comes from "Receipts," not "Fluency": The experts didn't care if the AI spoke perfectly or sounded like a human. They cared if they could verify the source. If an answer was smooth but had no page numbers, the experts immediately distrusted it. If it had a specific page citation, they trusted it, even if the writing was simple.
• They want a "Thinking Partner," not a "Magic 8-Ball": The experts didn't want the AI to give them a single, final answer. They wanted the AI to help them think. They preferred answers that showed different angles, trade-offs, and evidence so they could do their own reasoning. They treated the AI as a starting point for exploration, not the final verdict.
• Small Mistakes Break Trust: If the AI got a small detail wrong—like calling a state a city, or mixing up two different cities—it destroyed the expert's confidence in the entire answer. In high-stakes work, one error makes the whole thing look unreliable.
• More Info isn't Always Better: Sometimes, the AI gave very long, detailed answers that sounded impressive. But if those answers included a tiny bit of made-up context or over-generalized, the experts rated them lower than shorter, safer answers. They preferred "partial but true" over "complete but risky."

The Takeaway

The paper concludes that to make AI useful for serious work like saving the planet, we need to stop trying to make AI sound like a perfect human. Instead, we should design AI to be a transparent, accountable assistant.

Think of it this way: You don't want a lawyer who gives you a confident speech but hides their evidence. You want a lawyer who hands you the file, points to the exact page, and says, "Here is the evidence. Now, let's decide what to do." That is what SpheriCity aims to be for sustainability experts.

Technical Summary

Technical Summary: SpheriCity – Designing Trustworthy Conversational AI for Sustainability Decision Support

1. Problem Statement

Sustainability experts face significant cognitive and interactional burdens when navigating large, heterogeneous corpora of city-level circularity assessment reports. These documents, often exceeding 80–100 pages, contain fragmented information regarding material flows, infrastructure, and policy interventions. Traditional methods of cross-document synthesis require manual searching, reading, and cross-referencing, which is time-consuming and prone to missing critical connections.

While Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) systems offer the potential for rapid knowledge synthesis, they introduce critical risks in high-stakes sustainability contexts. Specifically, opaque reasoning, hallucinated facts, and a lack of source transparency undermine user trust and interpretability. Current conversational AI evaluations often rely on generic metrics (fluency, grammatical correctness) that fail to capture the specific needs of experts who require verifiable, unbiased, and traceable evidence to justify policy decisions. There is a gap in understanding how to design conversational agents that support "trustworthy knowledge sensemaking" rather than simply generating fluent answers.

2. Methodology

2.1 System Design: SpheriCity

The authors developed SpheriCity, a provenance-first conversational prototype designed to support knowledge sensemaking over a corpus of 36 circularity assessment reports (approx. 2,800 pages). The system architecture employs a RAG framework with three distinct retrieval configurations to test how evidence grounding influences trust:

1. Vector-based Retrieval: Uses embedding-based similarity search (all-MiniLM-L6-v2) with recursive character splitting and cross-encoder re-ranking to retrieve semantically similar passages.
2. Graph-based Retrieval: Utilizes the LightRAG system to construct a knowledge graph of entities (cities, policies, materials) and relationships, enabling multi-hop traversal of connected concepts.
3. Hybrid Retrieval: Generates independent responses from both vector and graph pipelines, then uses a synthesis prompt to integrate the strongest points, note discrepancies, and prioritize evidence-specific claims.

Key Design Principles:

• Provenance as Core: The interface explicitly separates source attribution from generation. Page numbers are derived from preserved document metadata rather than generated by the LLM, reducing the risk of hallucinated citations.
• Structured Synthesis: Responses are formatted as structured bullet points (interventions, outcomes, challenges) rather than narrative text to facilitate scanning and verification.
• Interaction Scaffolds: The interface includes prompt templates for common tasks (cross-city comparison, policy summarization) to guide query formulation.

2.2 Evaluation Framework

The study utilized a participatory design approach. Six sustainability experts co-defined an evaluation framework consisting of five dimensions, measured via a 15-item Likert scale:

1. Topical Relevance: Directness to the question.
2. Contextual Fit: Appropriateness to geographic/institutional scope.
3. Perceived Accuracy: Factual correctness based on expert knowledge.
4. Neutrality: Absence of bias or unwarranted perspective.
5. Depth of Information: Appropriate level of detail without speculation.

2.3 Study Protocol

A formative expert review was conducted with six practitioners. The study employed a static review protocol where experts rated pre-generated responses from the three retrieval configurations for 13 representative queries (covering city-specific, comparison, and recommendation tasks). This design allowed for controlled, within-rater comparisons of retrieval strategies while minimizing the cognitive burden of live interaction. Data collection included quantitative ratings and qualitative written reflections.

3. Key Results

3.1 Qualitative Findings

Thematic analysis of expert feedback revealed four critical insights:

• Trust Requires Provenance: Visible source attribution (report titles, page numbers) is a prerequisite for trust. Responses lacking traceable evidence were treated with skepticism, regardless of fluency. Experts viewed the absence of citations as a credibility disqualifier.
• Sensemaking Over Definitive Answers: Experts valued structured synthesis that supported reasoning and exploration over single, definitive answers. They preferred tools that helped them navigate trade-offs rather than providing automated recommendations.
• Interpretability Breakdowns Trigger Distrust: Small contextual errors (e.g., misidentifying a state as a city) disproportionately eroded trust. In high-stakes domains, such errors signaled a risk of overgeneralization, leading experts to reject the entire response.
• Workflow Misalignment: Experts viewed the chatbot as a starting point for exploration, not a final output. They expressed a need for features supporting annotation, note-taking, and justification logging to integrate AI outputs into their actual decision workflows.

3.2 Quantitative Findings

Linear mixed-effects analysis of 3,510 ratings indicated:

• Retrieval Performance: Vector-based and Hybrid retrieval approaches generally received higher ratings than Graph-based retrieval across all dimensions, particularly in Perceived Accuracy and Depth of Information.
• Task Sensitivity: The performance gap was most pronounced in city-specific queries, suggesting vector retrieval better captures localized contextual information.
• The Completeness-Trust Tension: While vector-based retrieval produced more comprehensive responses, experts remained sensitive to subtle hallucinations or contextual overgeneralizations within those responses. This confirms that increased depth does not automatically equate to increased trust if interpretability is compromised.

4. Key Contributions

The paper makes three primary contributions:

1. SpheriCity Prototype: A provenance-first conversational agent designed specifically for sustainability knowledge sensemaking, featuring structured synthesis and page-level traceability.
2. Expert-Grounded Evaluation Framework: A domain-specific evaluation instrument co-developed with sustainability experts, moving beyond generic metrics to assess relevance, contextual fit, accuracy, neutrality, and depth.
3. Design Insights for High-Stakes AI: Empirical evidence that in sustainability contexts, trust is calibrated through visible provenance and uncertainty communication rather than response fluency. The study highlights that AI systems should function as collaborative sensemaking tools rather than authoritative decision-makers.

5. Significance and Claims

The authors position this work as a necessary step toward responsible AI in sustainability. They claim that for conversational AI to be useful in high-stakes decision support, it must align with expert workflows that prioritize verification and accountability.

The paper modestly asserts that its findings suggest a shift in design priorities:

• From Fluency to Provenance: Systems must expose the provenance and scope of claims to enable appropriate reliance.
• From Answer Generation to Sensemaking: Interfaces should support iterative exploration, comparison, and evidence justification.
• From Automation to Collaboration: AI should augment human judgment by surfacing uncertainty and contextual constraints, rather than attempting to replace expert reasoning.

The authors acknowledge limitations, including the small sample size, the static nature of the evaluation (lacking multi-turn interaction), and the restriction to circularity reports. They frame these not as failures but as open research questions regarding how trustworthy AI should behave when incorrect answers have civic or environmental consequences. The work concludes that the future of AI in sustainability lies in designing systems that support transparent, accountable, and collaborative human-AI knowledge work.