💬 NLP

ArkTS-CodeSearch: A Open-Source ArkTS Dataset for Code Retrieval

This paper introduces ArkTS-CodeSearch, the first large-scale open-source dataset and benchmark designed to advance code intelligence and retrieval research for the ArkTS programming language.

Original authors: Yulong He, Artem Ermakov, Sergey Kovalchuk, Artem Aliev, Dmitry Shalymov

Published 2026-02-10

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Yulong He, Artem Ermakov, Sergey Kovalchuk, Artem Aliev, Dmitry Shalymov

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to find a specific recipe in a massive, disorganized library. You know you want something like "a spicy pasta dish with garlic," but the library doesn't have a search bar. Instead, you have to walk through the aisles, reading every single handwritten note stuck to the covers of cookbooks, hoping one of them matches your craving.

This is exactly the problem developers face with ArkTS, a new programming language used to build apps for smart devices (like the ones in the OpenHarmony ecosystem). Because ArkTS is relatively new, there isn't a "Google" for it. There are no organized datasets to help AI learn how to understand it, and no way to easily search through code using plain English.

The researchers behind this paper have essentially built the "Ultimate Search Engine Blueprint" for this language. Here is how they did it, broken down into three simple steps:

1. The Great Digital Scavenger Hunt (Data Collection)

The researchers went on a massive hunt across the internet (specifically sites like GitHub and Gitee). They weren't just looking for code; they were looking for "The Perfect Pair."

Think of it like finding a photo and its caption. They looked for a piece of code (the "photo") and the human-written comment explaining what it does (the "caption"). They used a specialized tool (called a "parser") to act like a high-tech magnifying glass, ensuring they only grabbed clean, well-structured snippets. They ended up with over 24,000 of these perfect pairs.

2. Teaching the "Brain" to Understand Context (Fine-Tuning)

Even if you have the data, you need a "brain" (an AI model) to understand it. Most existing AI brains were trained on common languages like Python or Java. Using them for ArkTS is like asking a world-class French chef to cook a very specific, rare dish from a remote island—they might get the idea, but they’ll struggle with the unique spices.

To fix this, the researchers took existing AI models and gave them "specialized training." They used a clever two-step method:

Step A: They taught the AI TypeScript (a "cousin" language to ArkTS) so it could learn the basic family traits.
Step B: They then taught it ArkTS specifically.

It’s like teaching a student Spanish first, and then teaching them a specific regional dialect. It makes the learning process much faster and more accurate.

3. The "Matchmaker" Test (Evaluation)

Finally, they tested how good these AI "brains" were at matchmaking. They gave the AI a description (e.g., "a function that calculates user age") and saw if the AI could find the exact piece of code that does that.

They used several "scorecards" (metrics) to see how often the AI:

Put the right answer at the very top of the list.
Put the right answer somewhere in the top five.

Why does this matter to you?

Even if you aren't a programmer, this is a huge step forward. As our world fills with "smart" everything—watches, cars, home appliances—the software running them needs to be smarter and more reliable.

By creating this dataset and these models, these researchers have laid the foundation for AI Coding Assistants (like a "Grammarly for Code") specifically for the next generation of smart devices. This means developers can work faster, make fewer mistakes, and build better technology for everyone.

Technical Summary: ArkTS-CodeSearch

ArkTS-CodeSearch is a research paper that introduces the first large-scale, open-source dataset and standardized benchmark specifically designed for code retrieval in ArkTS, the core programming language of the OpenHarmony ecosystem.

1. The Problem: The "Representation Gap"

While significant progress has been made in code semantic search for mainstream languages (Python, Java, JavaScript) via benchmarks like CodeSearchNet, the ArkTS ecosystem is severely underrepresented in current AI research.

Lack of Data: There are no publicly available, large-scale datasets for ArkTS.
Language Specificity: ArkTS, while derived from TypeScript, contains unique syntactic and semantic constructs (e.g., declarative UI annotations, distributed primitives) that general-purpose models struggle to capture.
Bilingual Complexity: A significant portion of the ArkTS community uses Chinese documentation, creating a linguistic mismatch for English-centric models.

2. Methodology

The authors propose a comprehensive pipeline for dataset construction and model optimization:

Data Collection & Extraction:
- Sources: They crawled repositories from both GitHub and Gitee (to capture both global and OpenHarmony-specific developer activity), resulting in 1,577 unique repositories.
- Parsing: Using tree-sitter-arkts, they parsed source files into Abstract Syntax Trees (ASTs) to accurately extract function-level code units paired with their natural language docstrings.
Task Definition: They established a single-search task (docstring-to-function retrieval) following the CodeSearchNet protocol.
Retrieval Approaches:
- Lexical Baseline: BM25 using the jieba tokenizer (to handle Chinese segmentation).
- Semantic Search: Dense vector retrieval using shared encoders to map docstrings and code into a unified embedding space, measured via cosine similarity.
Fine-Tuning Strategy: To bridge the domain gap, they explored three training configurations using supervised contrastive learning (Multiple Negatives Ranking Loss):
1. ARKTS-TRAINING: Fine-tuning on ArkTS data only.
2. TS-TRAINING: Fine-tuning on a large TypeScript corpus.
3. TS $\rightarrow$ ARKTS-TRAINING: A two-stage strategy (adapting to TypeScript first, then fine-tuning on ArkTS) to leverage the structural similarity between the two languages.

3. Key Contributions

The ArkTS-CodeSearch Dataset: A high-quality collection of 24,452 function-docstring pairs.
Standardized Benchmark: The first systematic evaluation framework for ArkTS code search using metrics like MRR, NDCG, and Recall@K.
Fine-tuned Models: The release of domain-specific embedding models (notably based on embeddinggemma) to the research community via HuggingFace.

4. Results and Key Findings

Model Performance:
- Zero-shot limitations: Off-the-shelf models perform poorly on ArkTS. The best zero-shot performer was Qwen3-Embedding-0.6B.
- The Power of Fine-tuning: Domain-specific fine-tuning significantly outperforms zero-shot baselines. Notably, the embeddinggemma-300m model, when fine-tuned, outperformed much larger models, proving that domain adaptation is more critical than model scale.
- Two-Stage Success: The TS $\rightarrow$ ARKTS strategy generally yielded the best results for larger models, demonstrating that pre-adapting on a related language (TypeScript) provides a superior semantic foundation.
Linguistic Insights: Models with strong Chinese language representations (e.g., bge-base-zh) significantly outperformed English-centric models, confirming the bilingual nature of the dataset.
Lexical vs. Semantic: BM25 performed poorly compared to neural embeddings, proving that simple keyword matching cannot capture the semantic nuances of ArkTS code.

5. Significance

This work provides the essential infrastructure required to advance AI-assisted software development for OpenHarmony. By providing a benchmark and a specialized dataset, the authors enable future research into automated code migration, defect detection, and intelligent developer tools for an emerging, high-growth ecosystem.