💬 NLP

Llettuce: An Open Source Natural Language Processing Tool for the Translation of Medical Terms into Uniform Clinical Encoding

This paper introduces Llettuce, an open-source, GDPR-compliant natural language processing tool that utilizes large language models and fuzzy matching to automate and improve the mapping of informal medical terms to OMOP standard concepts, addressing the limitations of existing manual-heavy solutions.

Original authors: James Mitchell-White, Reza Omdivar, Benjamin Partridge, Esmond Urwin, Karthikeyan Sivakumar, Ruizhe Li, Andy Rae, Xiaoyan Wang, Theresia Mina, Tom Giles, Diego Garcia-Gil, Tim Beck, John Chambers, Gra

Published 2026-03-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: James Mitchell-White, Reza Omdivar, Benjamin Partridge, Esmond Urwin, Karthikeyan Sivakumar, Ruizhe Li, Andy Rae, Xiaoyan Wang, Theresia Mina, Tom Giles, Diego Garcia-Gil, Tim Beck, John Chambers, Grazziela Figueredo, Philip R Quinlan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to organize a massive, chaotic library where every book has a different title, written in a different language, with some titles misspelled, and others written in slang. Now, imagine you need to sort all these books into a single, perfect, standardized catalog system used by doctors and researchers worldwide.

That is essentially the problem this paper solves, but instead of books, we are talking about medical terms.

Here is the story of Lettuce, the new tool designed to tidy up this mess.

The Problem: The "Lost in Translation" Crisis

In the world of medical research, data is gold. But for data to be useful, it needs to be FAIR (Findable, Accessible, Interoperable, Reusable). To do this, researchers use a standard system called OMOP, which is like a universal dictionary for medical concepts.

However, real-world medical data is messy.

A patient might write "Now Foods omega-3" on a questionnaire.
A doctor might type "Tylenol" in a note.
A nurse might write "Motrin" or even a typo like "Advilz".

The universal dictionary (OMOP) doesn't have "Now Foods omega-3." It has "Fish Oil." It doesn't have "Tylenol"; it has "Acetaminophen."

The Old Way:
Previously, researchers used tools like Athena (a search engine) or Usagi (a matching robot).

Athena is like using a library card catalog that only searches for exact words. If you search for "Fish Oil," it finds it. If you search for "Now Foods omega-3," it gets confused and suggests "Calcium tablets by Now Foods" because it sees the word "Now." It misses the meaning.
Usagi is a bit smarter, but it's like a spell-checker that only looks at how words are spelled. It struggles with slang, brand names, or typos. It often requires a human to sit there and manually fix the mistakes, which is slow and boring.

The Solution: Enter "Lettuce"

The authors built Lettuce, an open-source tool that acts like a super-smart, bilingual translator who understands not just words, but ideas.

Lettuce uses three main "superpowers" (or pathways) to find the right medical term:

1. The Keyword Detective (Text Search)

This is the old-school method. It looks for words that match exactly.

Analogy: If you are looking for a "Red Car," this tool finds "Red Car." But if you say "Crimson Sedan," it might miss it. It's fast, but a bit literal.

2. The Meaning Matcher (Vector Search)

This is where the magic happens. Lettuce uses AI to understand the concept behind the words. It turns words into mathematical coordinates (vectors).

Analogy: Imagine a giant map where words are cities. "Paracetamol" and "Acetaminophen" are two different names, but on this map, they are located in the exact same neighborhood because they mean the same thing. Even though "Fish Oil" and "Omega-3" look different, the AI knows they live next door to each other.
This allows Lettuce to say, "Ah, 'Now Foods omega-3' is actually just 'Fish Oil'!" even though the words don't match.

3. The Wise Librarian (LLM + RAG)

Sometimes, the map isn't enough. This is where Large Language Models (LLMs) come in. Think of the LLM as a brilliant librarian who has read every medical book in the world.

The Trick: The librarian doesn't just guess; they use RAG (Retrieval-Augmented Generation). Before the librarian answers, Lettuce hands them a stack of the top 10 most likely candidates found by the "Meaning Matcher."
Analogy: Instead of asking the librarian, "What is this?" in a vacuum, you say, "Here are 10 possibilities I found. Based on these, what is the best answer?" The librarian uses that context to give a much smarter, more accurate answer.

Why is Lettuce Special?

It's Private: Many AI tools (like the public version of ChatGPT) send your data to the cloud. In healthcare, patient data is sensitive (like a diary of your health). Lettuce is designed to run locally on your own computer or server. It never sends your patient data to a big tech company. It keeps the secrets safe.
It's Open: It's free and open-source, meaning anyone can look at the code, improve it, or use it without paying a fee.
It's Smarter: In tests, Lettuce found the correct medical term in the top 10 suggestions twice as often as the old tools.

The Results: A Real-World Test

The researchers tested Lettuce on two groups of data:

Formal Data: Drugs from a hospital trial (where names were mostly correct). Lettuce did well, but the old tools were okay here too.
Messy Data: Self-reported surveys from Singapore where people wrote down their meds in their own words (brands, slang, typos).
- The Old Tools: Struggled badly. They couldn't figure out that "Now Foods omega-3" was "Fish Oil."
- Lettuce: Succeeded brilliantly. By combining the "Meaning Matcher" with the "Wise Librarian," it correctly identified the standard medical term in nearly 50% of the messy cases (compared to much lower rates for other tools).

The Bottom Line

Lettuce is like a high-tech translator that bridges the gap between how humans talk about their health (messy, informal, brand-name heavy) and how computers need to store that data (clean, standardized, precise).

It makes medical research faster, more accurate, and safer for patient privacy, ensuring that when scientists look for data, they aren't searching for needles in haystacks, but finding the right answers in a perfectly organized library.

1. Problem Statement

The conversion of observational healthcare data into the OMOP Common Data Model (CDM) is critical for making data FAIR (Findable, Accessible, Interoperable, Reusable). However, mapping "source terms" (informal, unstructured, or brand-specific medical terms found in Electronic Health Records or patient questionnaires) to standardized OMOP concepts is a complex, labor-intensive process.

Existing solutions face significant limitations:

Athena Search: While comprehensive, it relies on manual exploration and struggles with the semantic nuances of informal language, often returning overwhelming or irrelevant results due to vocabulary overlap.
Usagi: A semi-automated tool using string-matching algorithms (e.g., BM25). It fails when source terms are informal, contain typos, or use brand names that do not lexically match the standard concept (e.g., mapping "Now Foods omega-3" to "Fish oil"). It requires extensive manual review and is prone to human error.
Proprietary LLMs: While Large Language Models (LLMs) offer semantic understanding, using cloud-based proprietary tools (e.g., OpenAI) raises critical concerns regarding GDPR compliance, data privacy, and the reproducibility of results when handling sensitive patient data.

2. Methodology: The Lettuce Architecture

Lettuce is an open-source (MIT License), local-deployable NLP tool designed to automate and enhance the mapping of medical terms to OMOP concepts. It is built on a modular architecture comprising an OMOP-CDM database and three distinct retrieval pathways:

A. Database & Pre-computation

Utilizes standard OMOP vocabularies (from OHDSI Athena).
Extends the schema with pre-computed lexical features (for keyword search) and dense vector embeddings (using PGVector) to support efficient similarity search.

B. Three Retrieval Pathways

Text Search (Lexical):
- Uses keyword matching after lemmatization (reducing words to root forms).
- Best for source terms that are lexically similar to existing OMOP concepts.
Vector-Based Semantic Search:
- Uses deep learning models (bidirectional encoders like BGE-small and PubMedBERT) to convert text into numerical embeddings.
- Captures semantic similarity even when wording differs (e.g., "paracetamol" vs. "acetaminophen").
- Performs exact similarity search to ensure accuracy, filtering results by domain or vocabulary.
Retrieval-Augmented Generation (RAG):
- Combines semantic search with an LLM.
- Process: If semantic search yields a low-confidence match, the top- $k$ relevant concepts are retrieved and inserted into an LLM prompt as context.
- Prompt Engineering: The system uses few-shot learning (examples of informal-to-formal pairs) and instructs the LLM to suggest a formal name. The LLM output is then validated against the OMOP vocabulary via a secondary keyword search to ensure the suggestion is a valid concept.

C. Interfaces

CLI: Command-line interface for direct execution.
HTTP API: Built with FastAPI, allowing integration with GUIs and other systems.
Lettuce-Inspector: A modular experimental framework used to benchmark pipelines, configure retrieval strategies, and evaluate performance against gold-standard annotations.

3. Key Contributions

GDPR-Compliant Automation: Provides a fully local, open-source solution that eliminates the need to send sensitive patient data to external cloud APIs, addressing a major barrier in healthcare data harmonization.
Hybrid NLP Pipeline: Successfully integrates lexical search, vector embeddings, and LLMs (RAG) to handle a spectrum of data quality from formal to highly informal.
Modular & Extensible Design: The architecture allows for the swapping of embedding models, LLMs, and retrieval strategies without rewriting the core system.
Comprehensive Evaluation Framework: Introduces lettuce-inspector to systematically test pipeline configurations, resource consumption, and accuracy.

4. Results

The tool was evaluated on two datasets:

Nottingham University Hospitals (NUH): 218 drugs from clinical trials (formal vocabulary).
HELIOS Study: Self-reported medication data from Singapore (informal, brand names, typos).

Key Findings:

Semantic vs. Lexical Search: Semantic search consistently outperformed lexical search (BM25).
- On the HELIOS dataset (informal), BM25 retrieved the correct concept in the top 10 only 10.25% of the time, whereas semantic search (BGE-small) achieved 22.25%.
- On the NUH dataset, BM25 achieved 50.9%, while semantic search reached 79.8%.
LLM & RAG Performance:
- LLMs alone struggled with accuracy on the HELIOS dataset (ranging 26.5%–35%).
- RAG significantly improved performance. The odds of an LLM providing the correct answer increased by 62% (Odds Ratio 1.62, $P < .001$ ) when RAG was enabled.
- Best Configuration: The Gemma 3 (12B) model combined with RAG using BGE-small embeddings and concept attributes (domain, vocabulary, class) in the prompt achieved the highest accuracy: 45.5% on HELIOS and 89.91% on NUH.
Context Matters: Providing the LLM with concept attributes (domain/vocabulary) in the prompt yielded better results than just providing concept names or scores.

5. Significance

Scalability: Lettuce offers a scalable solution for converting massive, messy real-world datasets into standardized formats, reducing the manual burden on data engineers and clinicians.
Data Governance: By running locally, it solves the critical issue of data sovereignty and privacy, making it viable for hospitals and research institutions with strict compliance requirements.
Future-Proofing: The agentic design allows for future integration of hierarchical classifiers and specialized biomedical models, ensuring the tool evolves with the OMOP ecosystem.
Reproducibility: The open-source nature and the lettuce-inspector framework promote transparent, reproducible research in healthcare data harmonization.

In conclusion, Lettuce represents a significant advancement in medical data standardization, proving that a locally deployed, RAG-enhanced NLP pipeline can significantly outperform traditional string-matching tools, particularly when dealing with the ambiguity of real-world patient-reported data.