Building Korean linguistic resource for NLU data generation of banking app CS dialog system

This paper presents the construction of the Financial Annotated Dataset (FIAD), a Korean linguistic resource derived from banking app reviews and Local Grammar Graphs, which is used to generate annotated training data that significantly improves the performance of various NLU models in banking customer service dialog systems.

The Gist

Imagine you are trying to teach a robot how to talk to people who are angry or confused about their bank accounts. To do this, the robot needs a "textbook" full of examples of what people actually say. But here's the problem: real people are messy. They use slang, they get angry, they use different levels of politeness, and they say the same thing in a thousand different ways. Collecting enough real examples by hand is like trying to catch every single drop of rain in a storm with a bucket—it takes forever and is incredibly expensive.

This paper introduces a solution called FIAD (Financial Annotated Dataset). Think of FIAD not as a bucket of rain, but as a high-tech "sentence factory."

Here is how the factory works, broken down into simple steps:

1. The Blueprint (Data Analysis)

First, the researchers didn't just guess what people say. They went to the "source": they looked at over 126,000 reviews of banking apps. They focused on the unhappy reviews (low scores) because that's where people are most likely to say, "Fix this!" or "I can't do that!" They used a computer tool to chop these reviews down into their smallest building blocks (words and grammar bits) to see what patterns emerged.

2. The Three Conveyor Belts (Resource Construction)

Instead of writing sentences one by one, they built a machine with three main conveyor belts. Each belt adds a specific part to the sentence:

• Belt A: The "What" (TOPIC)
  This belt holds the nouns. It has two bins:

  • Entities: Specific names like "Kakao Bank" or "Toss App."
  • Features: General banking words like "loan," "account," or "speed."
  • Analogy: This is like a box of Lego bricks. You can pick a red brick (Kakao Bank) or a blue brick (Toss App), but they are all the same shape (a noun).

• Belt B: The "Action" (EVENT)
  This belt holds the verbs and the logic. It decides what action is happening, like "create," "send," or "buy."

  • The Smart Filter: This belt is smart. It knows that you can "create" an account, but you can't "create" a speed. It checks the rules to make sure the action matches the noun. If you try to put "create" next to "speed," the machine rejects it.

• Belt C: The "Tone" (DISCOURSE MARKER)
  This is the most unique part. In Korean, how you end a sentence changes its meaning and politeness level. This belt adds the "flavor."

  • It can add a polite ending ("Could you please...?"), a direct command ("Do it!"), or a question ("Can you...?").
  • It also handles honorifics (respect levels). Just as you might speak differently to your boss versus your best friend, this belt can generate sentences that are formal, polite, or casual.

3. The Assembly Line (Data Generation)

Now, the magic happens. The machine connects these three belts.

• It picks a noun from Belt A.
• It picks a matching action from Belt B.
• It wraps it all in a specific tone from Belt C.

Because the machine can mix and match these parts in millions of ways, it can generate 60 trillion possible sentences! However, the researchers don't use all of them. They use a formula to pick the most natural-sounding, shorter sentences first (because people usually try to be brief).

4. The Test Drive (Experiments)

The researchers took the sentences generated by this factory and used them to train an AI model (a digital brain) to understand banking requests.

• The Result: The AI learned very well. It could correctly guess what the user wanted (the "Intent") about 95% of the time and could correctly identify the specific details (the "Entity," like which bank or which product) about 86% of the time.
• The Comparison: They tested different "brains" (pre-trained models) to see which one worked best with this new data. The model using a specific Korean language brain (KorBERT) performed the best.

The Bottom Line

The paper claims that instead of hiring hundreds of people to write thousands of sentences by hand, you can build a linguistic recipe book (FIAD). This book contains the rules of grammar, the vocabulary of banking, and the rules of politeness. By following these rules, you can automatically bake a massive, high-quality "cake" of training data. This allows you to teach a banking chatbot to understand Korean customers quickly, cheaply, and accurately, without needing to wait for real humans to type out every single variation of a request.

Technical Summary

Technical Summary: Building Korean Linguistic Resource for NLU Data Generation of Banking App CS Dialog System

Problem Statement
Task-oriented dialogue (TOD) systems rely heavily on Natural Language Understanding (NLU) to classify user intents and fill semantic slots. However, developing robust NLU models for the Korean language, particularly within the banking customer service (CS) domain, is hindered by a scarcity of publicly available annotated training data. Existing resources are often English-centric, privacy-restricted, or generated via crowdsourcing, which introduces noise and lacks the necessary linguistic diversity. Furthermore, Korean request utterances exhibit complex patterns involving specific discourse markers, honorifics, and performative predicates that are difficult to capture without domain-specific linguistic resources. The core challenge is generating a large-scale, high-quality, and linguistically diverse annotated dataset that reflects the specific syntactic and semantic constraints of Korean banking dialogues without incurring the prohibitive costs of manual annotation.

Methodology
The study proposes the construction of FIAD (Financial Annotated Dataset), a linguistic resource designed to generate annotated training data for banking CS dialog systems. The methodology follows a three-phase process:

1. Data Analysis: The authors analyzed a corpus of 126,598 banking app reviews (focusing on low-score reviews containing requests and complaints) to extract core keywords, nouns, predicates, and inflectional endings using TF-IDF weighting and the Mecab-Ko morphological analyzer.
2. Resource Construction: Based on the analysis, FIAD was structured into three modular components represented as Local Grammar Graphs (LGGs):
   • TOPIC: Divided into ENTITY (named entities like bank names and product names) and FEATURE (common nouns related to services like "loan" or "speed"). These modules define the slots to be filled.
   • EVENT: Contains predicative patterns (verbs) that invoke specific TOPIC modules based on semantic restrictions (e.g., "create" requires a "bank account" entity, not a "velocity" entity).
   • DISCOURSE-MARKER: Encodes multi-word expressions (MWEs) representing speech acts (Information Request, Action Request, Dissatisfaction, etc.), sentence types (declarative, imperative, interrogative, suggestive), and Korean honorific systems (six speech styles ranging from formal hapsyo-style to informal hay-style).
3. Data Generation: The three components are combined via Linking-LGGs to generate utterances. The system explores transitions in Finite-State Transducers (FSTs) compiled from the LGGs. To ensure naturalness and efficiency, a weighting formula prioritizes shorter utterances, and conjugation rules from the Dictionnaire Électronique du Coreen (DECO) are applied to verbs. This process allows for the generation of approximately 60 trillion potential utterance patterns, from which specific subsets can be generated based on desired size and linguistic characteristics.

Key Contributions

• FIAD Resource: The creation of a modular, LGG-based linguistic resource specifically for the Korean banking domain, capable of generating diverse, grammatically correct, and semantically annotated utterances.
• Linguistic Coverage: The explicit modeling of Korean-specific linguistic phenomena, including the agglutinative nature of the language, complex honorific systems, and diverse discourse markers for requests, which are often overlooked in standard template-based generation.
• Scalability and Flexibility: A system that allows researchers to adjust the generated dataset size and linguistic characteristics (e.g., politeness levels, specific entity types) by selecting specific modules, offering a time-efficient alternative to crowdsourcing or manual collection.

Experimental Results
The authors evaluated the utility of FIAD-generated data by training NLU models using the RASA framework and the DIET (Dual Intent Entity Transformers) classifier.

• Baseline Comparison: A DIET model trained on FIAD data (without pre-trained BERT embeddings) achieved an intent F1-score of 0.9142 and an entity F1-score of 0.8377. This outperformed a baseline DIET+BERT model trained on a different NLU benchmark dataset in intent classification, though entity extraction was slightly lower, attributed to the complexity of TOPIC combinations.
• Pre-trained Embeddings: Integrating pre-trained Korean BERT models significantly improved performance.
  • DIET+KorBERT achieved the highest intent extraction F1-score (0.95).
  • DIET+KoBERT achieved the highest entity extraction F1-score (0.86).
  • The results demonstrate that FIAD-generated data effectively trains models to extract intents and entities, with performance gains observed when combined with pre-trained embeddings.

Significance and Claims
The paper claims that FIAD provides a practical and efficient solution for constructing NLU training data in data-scarce domains. By leveraging Local Grammar Graphs to encode linguistic constraints and Korean honorifics, the resource generates vast amounts of "typical and grammatically fit" utterances with reduced workforce and time costs compared to traditional data collection methods. The study concludes that the modular structure of FIAD offers flexibility for building extensive training data tailored to specific aims, and the experimental results confirm that models trained on this synthetic data can effectively handle the semantic extraction tasks required for Korean banking customer service dialog systems.