Eliot: Interactively $\underline{E}$xploring Fast-Changing Scientific $\underline{Li}$terature Trends with $\underline{O}$nline Da$\underline{t}$a and Learning

Technical Summary: Eliot – Interactively Exploring Fast-Changing Scientific Literature Trends

Problem Statement
The rapid growth of scientific publishing makes tracking the evolution of fast-moving research areas increasingly difficult. While existing search engines and Large Language Model (LLM)-based assistants retrieve or summarize papers, they often obscure the underlying decisions regarding corpus selection, organization, and connection to temporal patterns. Current literature assistants frequently present ranked lists or opaque generated syntheses that hide the evidence behind trends. Researchers require an inspectable workflow to verify which documents support apparent temporal patterns, understand how papers separate into themes, and identify emerging directions without relying on fixed, expert-maintained taxonomies or domain-specific scripts.

Methodology
The authors present Eliot, a publicly deployed interactive system designed for the traceable exploration of evolving scientific literature using online data. The system operates on a query-time basis, retrieving papers from arXiv rather than relying on fixed benchmark datasets.

• System Pipeline:

  1. Retrieval: Users input explicit query terms (keywords/phrases), category filters, date ranges, and sorting criteria. The system retrieves relevant papers from arXiv, concatenating titles and abstracts for processing.
  2. Representation & Clustering: The retrieved corpus is represented using text embeddings. The system supports both automatic clustering (inferring structure) and user-controlled clustering (specifying the number of clusters).
  3. Labeling: Each cluster is assigned representative keywords using Class-based TF-IDF (c-TF-IDF), inspired by BERTopic, to summarize the distinguishing features of the group.
  4. Visualization: The interface displays cluster keywords, a temporal scatter plot showing the distribution of papers across publication years, and paginated lists of papers with source links.
  5. Traceability: A core design goal is auditability; users can inspect cluster memberships, compare labels, and access original source records rather than relying solely on summaries.

• Offline Configuration Evaluation (RQ1):
  To determine the optimal default configuration, the authors conducted an automated offline evaluation across eight diverse arXiv domains (e.g., Computer Science, Physics, Economics). They tested combinations of:

  • Document Representations: MiniLM, SciBERT, and TF-IDF.
  • Dimensionality Reduction: UMAP and SVD (with varying component counts).
  • Clustering Algorithms: K-Means, Agglomerative Clustering, and Fuzzy C-Means.
  • Metrics: Intrinsic clustering metrics (Silhouette Score, Calinski-Harabasz Index, Davies-Bouldin Index) and topic coherence metrics ($C_{V}$ and $C_{NPMI}$).

• User Evaluation (RQ2 & RQ3):
  The system's interpretability and utility were assessed via a scenario-based survey (10 participants) and an expert focus group (4 experienced researchers). Participants evaluated predefined scenarios (e.g., "Large Language Models" in CS, "Market and Prediction" in Quantitative Finance) and performed self-directed tasks.

Key Contributions

1. A Public Interactive System: Eliot combines query-time arXiv retrieval, unsupervised clustering, keyword labeling, temporal visualization, and document-level inspection into a single workflow.
2. A Traceable Workflow: The system exposes query settings, retrieved papers, cluster assignments, and source links, supporting auditability in exploratory literature analysis.
3. Empirical Configuration Analysis: An automated evaluation across eight domains identified a robust default configuration (MiniLM embeddings, 10-dimensional UMAP, and Agglomerative Clustering).
4. User-Centric Validation: Evidence from surveys and focus groups demonstrating that users find cluster labels meaningful and value the system for auditable overviews of rapidly changing technical areas.

Results

• Configuration Selection: The offline evaluation revealed that MiniLM embeddings were the top-performing representation across all eight domains, outperforming SciBERT and TF-IDF. UMAP was the preferred dimensionality reduction method, with 10 components being the most frequent optimal setting. Agglomerative Clustering appeared most frequently in top-ranked configurations, followed closely by K-Means. Consequently, the authors recommend MiniLM + 10D UMAP + Agglomerative Clustering as the practical default.
• User Interpretability: In the survey, 85% of responses rated cluster labels as meaningful (4 or 5 on a Likert scale), a result statistically significant ($p < 0.001$). While perceived usefulness of specific scenarios varied based on user background, participants showed a positive perception of the tool's utility for future research ($p = 0.047$).
• Expert Feedback: The focus group confirmed that Eliot is most valuable for providing traceable overviews of fast-changing areas. Users appreciated the ability to interact with visualizations and access source papers directly.
• Limitations Identified: Users noted potential confusion when automatic clustering produced different granularities across different time windows (due to fixed document fetch limits affecting embedding density). The authors addressed this by exposing controls to increase the maximum number of retrieved papers or switch to user-controlled clustering.

Significance and Claims
The paper claims that Eliot generalizes literature-evolution analysis beyond the constraints of hand-built taxonomies and domain-specific scripts used in previous studies (such as the LLM-planning literature analysis by Pallagani et al.). By shifting from supervised classification into fixed categories to unsupervised, query-time grouping, Eliot allows researchers to inspect the evidence behind literature trends directly.

The authors position Eliot not as a replacement for answer-oriented AI assistants (which synthesize claims) but as a complementary tool for the earlier exploratory phase. It helps researchers make the retrieved corpus visible as themes, labels, and temporal patterns, thereby supporting "auditable overviews" of rapidly changing technical areas. The work suggests that query-time clustering and temporal inspection can complement search and generation tools by enabling researchers to refine and verify the evidence behind emerging trends.

The authors remain modest in their claims, noting that the configuration results are "practical defaults" rather than universal optima, and that the user studies are preliminary, calling for larger-scale controlled evaluations in future work.