FreeTxt-Vi: A Benchmarked Vietnamese-English Toolkit for Segmentation, Sentiment, and Summarisation

FreeTxt-Vi is a free, open-source web toolkit that integrates a benchmarked bilingual Vietnamese-English NLP pipeline for segmentation, sentiment analysis, and summarization, enabling non-programmers to analyze text data while demonstrating competitive performance against existing baselines.

The Gist

Imagine you have a massive library of letters, emails, and survey responses written in two languages: Vietnamese and English. You want to understand what people are saying, how they feel, and what the main stories are. But there's a problem: the library is chaotic, the books are in different scripts, and the tools to read them usually require you to be a computer programmer.

FreeTxt-Vi is the new, free, user-friendly "magic library assistant" designed to solve this. It's a web-based toolkit that lets anyone—teachers, researchers, or community leaders—analyze text without needing to write a single line of code.

Here is how it works, broken down into simple concepts:

1. The Language Translator & Organizer (Segmentation)

The Problem: In English, words are like distinct Lego bricks separated by spaces (e.g., "cat sat"). In Vietnamese, the "bricks" are often glued together, and spaces separate syllables instead of whole words. For a computer, the word "student" might look like three separate pieces: "học" + "sinh". If the computer doesn't know how to glue them back together, it gets confused.

The Solution: FreeTxt-Vi uses a hybrid team to fix this.

• Think of VnCoreNLP as a skilled Vietnamese librarian who knows exactly how to glue the syllables back into proper words.
• Think of BPE (Byte Pair Encoding) as a smart robot that understands how modern AI speaks.
• Together, they act like a bilingual translator and organizer, ensuring that whether the text is in English or Vietnamese, the computer sees the words correctly before trying to analyze them.

2. The Mood Reader (Sentiment Analysis)

The Problem: You have thousands of feedback forms. Do people love the new school policy, or are they angry? Reading them one by one takes forever.

The Solution: FreeTxt-Vi has a super-smart mood ring.

• It uses a "trained brain" (a fine-tuned AI model) that has read millions of reviews in both languages.
• It doesn't just say "happy" or "sad." It can tell you if someone is "very happy," "mildly annoyed," or "neutral."
• It draws colorful charts (like pie charts and bar graphs) so you can instantly see the "emotional temperature" of your data. It's like having a weather forecast for people's feelings.

3. The Story Summarizer (Summarisation)

The Problem: Imagine a 50-page report full of messy, rambling feedback. You need the "gists" in 30 seconds.

The Solution: FreeTxt-Vi offers two ways to shrink the story:

• The Highlighter (Extractive): It acts like a student underlining the most important sentences in the original text. It's honest and shows you exactly what it picked.
• The Storyteller (Abstractive): This is the cool part. It uses a powerful AI (Qwen2.5) that acts like a creative journalist. Instead of just copying sentences, it reads the whole mess, understands the meaning, and writes a brand-new, short, fluent summary in its own words.
• The "Focus" Feature: You can even tell the AI, "Only summarize the parts about teachers," or "Only tell me about safety issues." It acts like a filter that only lets the specific topics you care about through.

4. The Visual Explorer (Word Clouds & Trees)

The Problem: Sometimes you just want to see what words pop up the most, or how words hang out together.

The Solution:

• Word Clouds: Imagine a cloud where the bigger the word, the more important it is. FreeTxt-Vi can show you these clouds based on simple counts, or based on "how unique" a word is compared to normal language.
• Word Trees: This is like a family tree for words. If you pick the word "Knowledge," the tool draws branches showing every word that usually comes before or after it. It helps you see patterns, like how "Knowledge" is often followed by "is power" or "is important."

Why Does This Matter?

For a long time, powerful AI tools were like Ferraris: fast and amazing, but only for people who knew how to drive them (programmers) and mostly built for English speakers.

FreeTxt-Vi is like a reliable, free family car that anyone can drive. It's built specifically to handle the unique challenges of the Vietnamese language while still working perfectly for English.

The Big Win:

• It's Free: No expensive subscriptions.
• It's Open: Anyone can look under the hood to see how it works.
• It's Fair: It treats Vietnamese and English as equals, giving a voice to a language that often gets ignored by big tech.

In short, FreeTxt-Vi turns a mountain of confusing text into a clear, colorful, and understandable story, helping people make better decisions based on what people are actually saying.

Technical Summary

Here is a detailed technical summary of the paper "FreeTxt-Vi: A Benchmarked Vietnamese–English Toolkit for Segmentation, Sentiment, and Summarisation."

1. Problem Statement

The paper addresses the significant gap between advances in Natural Language Processing (NLP) for Vietnamese and their practical application by non-specialists. While Vietnamese is spoken by over 80 million people, it remains an under-resourced language in NLP research compared to English.

• Linguistic Challenges: Vietnamese orthography separates syllables with spaces rather than words, creating complex word segmentation issues that standard tokenizers (like BPE or WordPiece) often handle poorly without linguistic guidance.
• Accessibility Gap: Existing high-performance Vietnamese NLP tools (e.g., VnCoreNLP, PhoBERT, ViT5) require programming expertise and familiarity with machine learning frameworks, making them inaccessible to researchers in social sciences, education, and humanities.
• Bilingual Limitations: Existing text analysis platforms (e.g., Voyant Tools, FreeTxt) generally lack support for Vietnamese or do not address the specific challenges of bilingual (Vietnamese-English) data processing, such as mixed-language datasets common in surveys and feedback forms.

2. Methodology

The authors developed FreeTxt-Vi, an open-source, web-based toolkit that integrates corpus linguistics methods with modern transformer-based NLP. The system is designed to process free-text data without requiring user programming skills.

Core Architecture & Pipeline

The toolkit utilizes a unified bilingual processing pipeline with the following key components:

• Data Preprocessing:
  • Language Detection: Automatically identifies dominant languages (Vietnamese or English) in mixed datasets.
  • Hybrid Segmentation: Combines VnCoreNLP (a rule-based/linguistic segmenter) with Byte Pair Encoding (BPE). VnCoreNLP handles Vietnamese word segmentation to preserve multi-syllabic words, while BPE handles subword tokenization for transformer compatibility. English text is processed using standard libraries (NLTK/spaCy).
• Sentiment Analysis:
  • Utilizes a fine-tuned TabularisAI multilingual transformer model.
  • Supports both coarse-grained (3-class) and fine-grained (5-class) sentiment classification.
  • The model is jointly fine-tuned on Vietnamese (VLSP 2016) and English (IMDb) datasets to create a unified bilingual classifier.
• Summarisation:
  • Extractive: Uses the TextRank algorithm to identify salient sentences.
  • Abstractive: Leverages a fine-tuned Qwen2.5 (7B) Large Language Model (LLM).
  • Innovation: Introduces aspect-guided summarisation, where the system detects salient themes (e.g., social, academic) and generates focused summaries based on user-selected aspects. It also supports prompt-guided summarisation.
• Corpus Linguistics & Visualization:
  • Includes interactive Word Trees (KWIC patterns), Concordances (with AI-generated lexical suggestions), and Word Clouds (Frequency, Log-likelihood, and Keyness).
  • For Vietnamese, visualizations rely on VnCoreNLP to ensure multiword expressions are treated as single tokens.

3. Key Contributions

1. Unified Bilingual Pipeline: The design of a single processing pipeline that effectively handles both Vietnamese and English, integrating linguistic segmentation (VnCoreNLP) with subword tokenization (BPE) and modern LLMs.
2. Novel NLP Components:
   • Release of a hybrid segmentation strategy (VnCoreNLP + BPE).
   • A bilingual sentiment classifier fine-tuned on both languages.
   • An aspect-guided abstractive summarisation module using Qwen2.5, a feature previously unavailable for Vietnamese NLP.
3. Comprehensive Evaluation: The paper provides a rigorous three-part evaluation (Segmentation, Sentiment, Summarisation) benchmarking the toolkit against widely used baselines.
4. Accessibility: By providing a web interface, the toolkit lowers technical barriers, enabling researchers in non-CS fields to analyze qualitative Vietnamese data at scale.
5. Open Source: All code, models, and resources are publicly available on GitHub to ensure reproducibility.

4. Experimental Results

The system was evaluated on standard datasets: VLSP2013 (Segmentation), VLSP2016/IMDb (Sentiment), and VNDS/CNN/DailyMail (Summarisation).

• Word Segmentation & Tokenization:
  • The hybrid approach achieved 98.1% F1 on Vietnamese (VLSP2013), competitive with the top-performing monolingual segmenter (NlpHUST at 98.3%) and significantly outperforming standard subword tokenizers like BPE (89.4%) and WordPiece (91.2%).
  • For English, it achieved 99.94% accuracy with the highest throughput (75,500 docs/sec) among linguistically informed baselines.
• Sentiment Analysis:
  • The fine-tuned bilingual model achieved 95.2% accuracy on Vietnamese (VLSP2016) and 95.0% on English (IMDb).
  • It outperformed the base TabularisAI model (+1.4% / +0.9%) and the best monolingual Vietnamese baseline, PhoBERT, by +1.8% on Vietnamese.
  • Inference speed remained competitive (~205–215 samples/sec).
• Summarisation:
  • The fine-tuned Qwen2.5 model achieved state-of-the-art results, with ROUGE-1/2/L scores of 53.1/27.5/48.4 on Vietnamese (VNDS) and 52.4/27.0/48.0 on English (CNN/DailyMail).
  • This represents a consistent improvement of +3 to +4 ROUGE points over strong baselines like Flan-T5 and Gemma-7B, demonstrating effective cross-lingual transfer.

5. Significance and Impact

• Bridging the Resource Gap: FreeTxt-Vi provides critical infrastructure for Vietnamese NLP, a language often overlooked in favor of high-resource languages. It transforms complex NLP tasks into accessible tools for domain experts.
• Multilingual Research: By validating a unified bilingual pipeline, the paper demonstrates that joint fine-tuning on Vietnamese and English improves low-resource performance without degrading high-resource performance, offering a blueprint for other under-resourced language pairs.
• Domain Applicability: The toolkit is immediately applicable to education, digital humanities, cultural heritage, and social sciences, where qualitative text data is abundant but difficult to process manually.
• Reproducibility: The open-source nature of the project promotes transparency and allows for the extension of these methods to other Southeast Asian languages.

In conclusion, FreeTxt-Vi represents a significant step forward in democratizing NLP for Vietnamese, combining rigorous linguistic preprocessing with state-of-the-art generative AI to create a robust, user-friendly, and evaluated analysis platform.