💻 computer science

Transparent Context-aware Retrieval for Mental Health and Substance Use Service Navigation

This study developed and evaluated a transparent, context-aware retrieval system for mental health and substance use services using the HealthLink BC directory, demonstrating that combining semantic search with explicit context weighting improves ranking performance over lexical baselines while highlighting the need for further validation with real-world data and human relevance judgments before clinical deployment.

Original authors: Nikesh Adhikari, Shankar Ghimire

Published 2026-09-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Nikesh Adhikari, Shankar Ghimire

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Finding the right help for mental health or substance use is often less about the services not existing and more about the difficulty of finding the one that fits a person's specific life. Imagine standing in a vast library where thousands of books are available, but the catalog is disorganized, the titles are vague, and you do not know which section holds the answer to your immediate problem. In British Columbia, Canada, a public directory lists over four thousand such services, ranging from crisis support to long-term counseling, yet navigating this list can feel just as overwhelming as the library itself. People face barriers like cost, long waiting times, and a lack of clarity about who is eligible for what. When a person is in distress, they need a way to cut through the noise and find a service that matches their location, their language, their age, and their specific needs. This is where the science of information retrieval steps in, not to diagnose a patient or prescribe a treatment, but to act as a sophisticated librarian that can sort through thousands of options to present the most relevant ones first.

Researchers at Coventry University set out to build and test a transparent system designed to do exactly this: rank mental health and substance use services based on a person's context. They used a public dataset containing 4,263 unique service records from the HealthLink BC directory. The goal was not to create a medical device that decides what is wrong with a person, but to create a navigation tool that understands the practical details of a person's situation. The team developed a computer program that could look at a service description and a user's request, then calculate how well they matched. They tested different ways of doing this matching. One method relied on simple word counting, looking for shared vocabulary between the request and the service description. Another method used a more advanced technique that understands the meaning behind words, recognizing that "anxiety" and "worry" are related even if the exact words differ. Finally, they added a layer that explicitly checked for practical details like whether the service accepts students, is located in a specific city, or offers wheelchair accessibility.

The researchers created a test scenario where they generated 200 simulated questions based on the actual service records themselves. For each question, they knew exactly which service record was the correct answer because they had written the question using that record's own details. They then asked their system to find that specific record among the thousands of others. When the system relied only on the meaning of the words, it successfully placed the correct service in the top ten results about 29 percent of the time. However, when the system was also told to pay attention to the specific context—such as the city, the language, and the audience—the success rate improved. The correct service appeared in the top ten results 32.6 percent of the time. While this might seem like a small increase, it represents a meaningful improvement in the system's ability to retrieve the right information when practical constraints are involved. The researchers also built a learning component that could be trained to weigh these factors, and in a controlled test using self-generated data, this component showed a very high ability to distinguish between a matching service and a random one.

Crucially, the system was designed to be transparent, meaning it does not hide its reasoning behind a black box. When it recommends a service, it can explain exactly why. It can state that a service was chosen because it is in the user's city, because it serves students, or because the description strongly matches the user's needs. This clarity is vital in a field where trust is essential. The researchers found that while the system could successfully match records to queries, it had clear limitations. The test relied on questions that were written using the answers themselves, which is a useful way to check if the system works technically, but it does not prove that the system would work well with real people asking real questions. The study explicitly ruled out the idea that this system is ready for clinical use. It cannot assess risk, determine if a person is in crisis, or replace a human navigator. The data it uses can also become outdated, and the system does not account for complex eligibility rules that might not be written in the service description.

The study concludes that while the technical foundation for a transparent, context-aware navigation tool is feasible, it is not yet a finished product for public deployment. The improvement seen in the tests proves that adding specific details like location and audience helps the system find the right records, but the current evaluation does not guarantee that a person in a real-world crisis would find the help they need. The researchers emphasize that before such a tool could be used to guide people to care, it would need to be tested with real help-seeking questions, evaluated by human experts for fairness and accuracy, and checked regularly to ensure the service information is current. The work serves as a proof of concept, demonstrating that a machine can be taught to understand the practical constraints of seeking help, but it also draws a firm line between a working algorithm and a trusted healthcare resource. The path forward requires more than just better code; it demands rigorous testing with the people who will actually use the system to ensure that the technology serves them safely and effectively.

Technical Summary: Transparent Context-aware Retrieval for Mental Health and Substance Use Service Navigation

Problem Statement

Accessing mental health and substance use services is hindered not only by the existence of services but by the difficulty of identifying appropriate, reachable, and acceptable options. Individuals face structural and informational barriers, including cost, waiting times, geographic dispersion, and gaps in culturally responsive care. While public service directories (such as the HealthLink BC directory) aggregate thousands of heterogeneous services, they often transfer the navigation burden to the user.

Current health recommender systems frequently rely on offline evaluations, lack end-user participation, and often fail to distinguish between clinical decision-making and service navigation. There is a specific gap in creating transparent, auditable systems that rank open directory records based on public taxonomy, eligibility, and practical context (e.g., location, language, student status) without using patient data or inferring diagnoses.

Methodology

This study conducted a retrospective computational feasibility analysis using the HealthLink BC Mental Health and Substance Use open dataset, comprising 4,263 uniquely identified service records. The system was designed as a non-diagnostic, transparent ranking pipeline.

Data Representation

Service Documents: A composite document was created for each record, concatenating taxonomy, audience, program description, language, location, and accessibility fields.
Contextual Features: Explicit binary and categorical fields were extracted for student eligibility, wheelchair accessibility, city, language, and audience.

Retrieval and Ranking Models

The study implemented and compared five distinct approaches:

Lexical Baselines:
- TF-IDF: Used unigrams and bigrams with sublinear term frequency.
- BM25: Employed probabilistic term-frequency saturation and document-length normalization.
Semantic Retrieval:
- Sentence-BERT (Bi-Encoder): Used all-MiniLM-L6-v2 to generate 384-dimensional sentence embeddings for cosine similarity matching.
Context-Aware Weighted Ranker:
- A linear combination of semantic similarity and explicit feature matches.
- Formula: $S = 0.50s_{sem} + 0.15s_{aud} + 0.10s_{stu} + 0.10s_{city} + 0.05s_{lang} + 0.10s_{access}$
- Matches were determined via substring or exact matching; if a user did not request a specific attribute (e.g., wheelchair access), its contribution was zero.
Learned Reranker:
- A Logistic Regression model trained on 1,000 weakly supervised pairs (500 self-matched positive pairs and 500 random negatives).
- Features included the six context/semantic scores used in the weighted ranker.
Optional Cross-Encoder:
- A ms-marco-MiniLM-L-6-v2 model was tested on a single example to rerank the top 30 semantic candidates, though no aggregate metrics were computed for this component.

Evaluation Protocol

Synthetic Queries: 200 queries were generated by concatenating fields (audience, taxonomy, city, language) from the source records themselves.
Ground Truth: The source record from which a query was derived was treated as the single relevant item.
Metric: Mean Reciprocal Rank at 10 (MRR@10) was the primary metric. Precision, Recall, and NDCG were defined but not executed as aggregate results.
Explainability: A rule-based generator exposed satisfied predicates (e.g., "City matches," "Student eligible") and semantic match strength thresholds.

Key Results

Retrieval Performance

Semantic Baseline: The Sentence-BERT bi-encoder achieved an MRR@10 of 0.2924.
Context-Aware Ranker: Adding explicit context features increased the MRR@10 to 0.3258.
- This represents an absolute increase of 0.0334 and a relative increase of 11.4%.
Interpretation: The improvement indicates that explicit structured fields helped recover the source record earlier when the evaluation profile was derived from that same record.

Learned Reranker Performance

The logistic reranker achieved 0.96 accuracy on a stratified 20% hold-out set.
Feature Coefficients: Location matching had the largest positive coefficient (5.859), followed by semantic similarity (2.420) and audience match (1.827). Notably, the accessibility coefficient was negative (-1.260), which the authors attribute to sampling artifacts and feature dependencies rather than a lack of value for accessibility.
Limitation: The high accuracy is interpreted as "weak-label discrimination" because the positive training pairs were constructed by copying information from the target record, creating a form of data leakage.

Explainability

The system successfully generated local explanations identifying which specific predicates (e.g., city, language, student status) were satisfied. However, the explanations did not display weight magnitudes, negative evidence, or uncertainty estimates.

Key Contributions

Reproducible Architecture: A transparent, content-and-context pipeline for ranking open mental health service directories without patient data.
Executed Comparison: A fixed-seed comparison demonstrating that explicit context improves early ranking (MRR@10) over pure semantic retrieval under a synthetic protocol.
Interpretable Layer: A direct mapping of ranking scores to understandable match reasons (predicates) rather than opaque post-hoc attributions.
Critical Validation Boundary: A clear distinction between technical feasibility (the system works as designed computationally) and clinical effectiveness (the system is safe and useful for real users), explicitly outlining the evidence required for responsible progression.

Significance and Claims

The paper modestly claims technical feasibility rather than clinical effectiveness. The authors emphasize that:

The system is a service navigation tool, not a clinical decision support system. It does not assess risk, diagnose, or select treatments.
The results are bounded by self-consistency; the queries and relevance labels were derived from the same records, meaning the system is tested on its ability to "find itself" rather than on real-world help-seeking queries.
Transparency is a primary feature: the architecture allows for the inspection of scoring logic, weights, and match reasons, which is critical for high-stakes domains.
Limitations: The study lacks independent human relevance judgments, user safety evaluations, and fairness audits. The "leakage" in the synthetic evaluation (using target record data to generate queries) limits the generalizability of the performance metrics.

The authors conclude that while the prototype demonstrates a viable architecture for transparent directory ranking, it requires expert-labelled benchmarks, real-world queries, and participatory evaluation with service users and navigators before it can be considered for deployment.