💻 computer science

Linting Style and Substance in READMEs

This paper introduces LintMe, a design probe that empowers users to create context-specific README linters by combining programmatic checks with LLM-based content evaluation, demonstrating through user studies that this hybrid approach effectively balances automated style and substance improvements with authorial agency.

Original authors: Hima Mynampaty, Nathania Josephine, Katherine E. Isaacs, Andrew M. McNutt

Published 2026-03-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hima Mynampaty, Nathania Josephine, Katherine E. Isaacs, Andrew M. McNutt

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you've just baked a delicious, complex cake. You've put in the effort, but when you hand it to a friend, they stare at it confused. They don't know if it's for breakfast or dessert, they can't find the recipe card, and they're not sure if it contains nuts (which might be an allergy).

In the world of software, that cake is a README file. It's the "instruction manual" or the "welcome mat" for a piece of code. If the README is messy, confusing, or missing key details, people won't use the software, no matter how good the code underneath is.

This paper introduces a new tool called LintMe (short for "Lint Me, please!"). Think of LintMe not just as a spell-checker, but as a super-smart, customizable sous-chef that helps you write better READMEs.

Here is the breakdown of what they did, using some everyday analogies:

1. The Problem: One Size Does Not Fit All

Currently, there are tools that check if your README has correct grammar or if your bullet points are aligned (like a basic spell-checker). But these tools are too simple.

The Analogy: Imagine a spell-checker that tells you your cake recipe is missing a comma. It doesn't tell you that you forgot to list the ingredients, or that you didn't say "Do not eat if you are allergic to peanuts."
The Reality: A README for a scientific dataset needs different things than a README for a video game. A spell-checker doesn't know the difference. It treats all documents the same.

2. The Solution: LintMe (The Customizable Sous-Chef)

The authors built LintMe, a system that lets you create your own "rules" for what makes a good README.

The Analogy: Instead of a rigid rulebook, LintMe is like a LEGO set for rules. You can snap together different blocks to build a checker that fits your specific needs.
How it works:
- The Bricks (Operators): These are pre-made tools. One brick counts how many emojis you used. Another checks if your links are broken. Another scans for offensive words.
- The Glue (LLMs): This is the "magic" part. They connected these bricks to Large Language Models (AI). This allows the tool to understand meaning, not just math.
- The Recipe: You can tell the AI, "Check if the tone is welcoming," or "Make sure there is a section explaining how to install this." The AI reads your text and checks against your custom rule.

3. The "Escape Hatches" (When the Bricks Aren't Enough)

Sometimes, the pre-made LEGO bricks aren't enough. Maybe you need to check something very specific, like "Does this code actually run?"

The Analogy: LintMe has a "Magic Wand" feature. If the standard rules don't work, you can write a tiny bit of custom code (JavaScript) or ask the AI to do a deep dive. It's like having a master chef step in to fix a complex sauce that the automated machine can't handle.

4. The User Study: Do People Like It?

The researchers tested LintMe with 11 people (mostly students and developers).

The Reaction: People loved the idea. They felt it helped them remember to add important sections they usually forget (like a "Table of Contents" or "How to Contribute").
The Hurdle: Some people found it a little tricky at first, like learning a new video game controller. They had to learn how to "snap" the rules together. But once they got the hang of it, they felt in control.
The "Agency" Factor: Crucially, LintMe doesn't just fix everything for you automatically. It points out issues and suggests fixes, but you decide whether to accept them. This prevents the "automation bias" where we blindly trust the computer even when it's wrong.

5. The "Recipe" Test: Does It Work Elsewhere?

To prove LintMe is flexible, the researchers tried using it on cooking recipes instead of code.

The Analogy: They asked LintMe to check if a recipe for "Lasagna" had all the right parts: Did it list the ingredients? Did it say how long to bake it? Did it use generic names (like "flour") instead of brand names (like "King Arthur Flour")?
The Result: It worked! It found errors in famous recipe websites that humans missed. This proves the tool isn't just for coders; it can check any kind of instruction manual, from game guides to legal documents.

6. Why This Matters

Before this, if you wanted to check if your documentation was "good," you had to hire a human expert or use a very basic tool that only checked spelling.

The Big Picture: LintMe bridges the gap between rigid computer rules (like checking line length) and human nuance (like checking if the tone is friendly).
The Future: It opens the door for tools that can check complex documents, ensuring that instructions are not just grammatically correct, but actually useful and safe for the people reading them.

In short: LintMe is a smart, customizable assistant that helps you write better instructions by letting you teach it exactly what "good" looks like for your specific project, using a mix of simple logic and advanced AI.

1. Problem Statement

README files are the primary interface for users interacting with software projects, yet their quality varies wildly. Existing documentation linters (e.g., markdownlint, proselint) are limited to syntactic and stylistic checks (e.g., heading hierarchy, grammar, list indentation). They fail to evaluate substantive content, such as:

Reproducibility details (e.g., does the example code actually run?).
Context-specific requirements (e.g., data provenance for datasets vs. quick-start guides for libraries).
Semantic issues (e.g., exclusionary language, ambiguity, or missing ethical considerations).
Accessibility elements (e.g., alt-text for images).

Furthermore, different communities have distinct standards. A rigid, one-size-fits-all linter cannot accommodate the diverse needs of research software, open-source libraries, and interactive systems. There is a need for a tool that allows users to define custom, context-aware rules that blend programmatic logic with semantic understanding.

2. Methodology: The LintMe System

The authors propose LintMe, a design probe and system that extends linting from style to substance using a Lightweight Domain-Specific Language (DSL).

Core Architecture

Operator-Based DSL: LintMe rules are defined as pipelines of composable operators.
- Predefined Operators: 21 built-in operators handle specific tasks like extract (finding AST nodes), count, regexMatch, threshold (checking limits), and fetchFromGithub (retrieving repo metadata).
- Escape Hatches: To handle complex or undefined requirements, the system includes "escape hatch" operators:
  - evaluateUsingLLM: Sends context to a Large Language Model (LLM) for semantic evaluation (e.g., checking tone or jargon).
  - customCode: Executes arbitrary JavaScript for bespoke logic or library reuse.
  - execute: Runs command-line instructions found in the README to verify installation steps.
Execution Model: Rules are executed as a pipeline. Operators process Markdown Abstract Syntax Trees (ASTs) generated by mdast. Data flows between operators (e.g., extract $\to$ count $\to$ threshold).
User Interface: A web-based sandbox (lintme.netlify.app) and CLI tool. The interface supports:
- Rule Authoring: Manual YAML editing, LLM-assisted rule generation (converting natural language to YAML), and visual rule construction.
- Feedback: Errors are highlighted directly in the text (wavy underlines) with specific blame attribution (line numbers or text chunks).
- Presets: Pre-configured rule sets for specific domains (Software Library, Interactive System, Dataset Repository).

Design Goals

The system was built around three goals derived from the "linting ladder" concept:

DGauthor (Authorable): Users must be able to create and modify rules without deep programming expertise.
DGcomm (Community-Tunable): Rules must be adaptable to specific community standards and contexts.
DGagency (Usability & Agency): The tool should assist, not dominate. It provides feedback and fixes but requires human confirmation to avoid automation bias.

3. Evaluation

The authors conducted three complementary studies to evaluate LintMe.

A. User Study (N=11)

Method: Participants (students and industry professionals) used LintMe to lint their own READMEs and create new rules.
Findings:
- Positive Reception: Users found the tool useful for identifying gaps and enforcing standards.
- Learning Curve: Editing YAML and understanding the operator pipeline presented a barrier, though LLM-assisted generation helped.
- Agency: Users appreciated the ability to selectively accept/reject fixes and customize rules, though they noted the risk of automation bias if "fix-all" buttons were too aggressive.
- Community Fit: Participants confirmed that different domains (e.g., ML vs. Visualization) require different documentation structures, validating the need for tunable presets.

B. Comparison with Naive LLM Usage

Method: The authors compared LintMe against direct LLM prompting (Free Prompt and Rules-Provided) using five diverse READMEs (Vega-Lite, TensorFlow, etc.) and four different LLM models.
Findings:
- Precision: LintMe identified significantly more errors ( $\mu=25.4$ ) than naive LLM usage ( $\mu=9.6$ for free prompts).
- Consistency: LLMs were inconsistent across runs and often missed specific structural or semantic violations that LintMe caught via its programmatic checks (e.g., missing alt-text, hate speech).
- Value of Curation: The structured nature of LintMe's rule set provided a more reliable "ground truth" for community standards than ad-hoc LLM prompts.

C. Expressivity Case Study (Culinary Recipes)

Method: The authors adapted LintMe to lint culinary recipes (a non-code domain) using a new preset of 12 rules (e.g., "no multi-task instructions," "temperature formatting").
Findings:
- LintMe successfully applied the same DSL operators to a completely different domain.
- It detected systematic errors in popular recipe sites (e.g., missing units, nested steps) that standard text linters would miss.
- This demonstrated the system's flexibility to evaluate "substance" beyond software documentation.

4. Key Contributions

Novel Linting Paradigm: A shift from purely syntactic linting to substantive linting, capable of evaluating content quality, reproducibility, and cultural norms.
Hybrid DSL Design: A unique architecture that combines programmatic operators (for deterministic checks like link validation) with LLM calls (for semantic checks like tone and jargon), bridging the gap between rigid code and flexible natural language.
End-User Rule Authoring: A system that empowers non-experts to define complex, context-specific documentation standards using a DSL augmented by LLMs.
Empirical Validation: Evidence that structured, rule-based linting outperforms naive LLM prompting in terms of error detection and consistency, while maintaining human agency.

5. Results and Significance

Effectiveness: LintMe successfully identifies a broader range of issues (style, structure, and substance) than existing tools or standalone LLMs.
Flexibility: The system is domain-agnostic, proven by its successful application to both software READMEs and culinary recipes.
Sociotechnical Impact: The work highlights that documentation quality is a sociotechnical issue. By allowing communities to encode their own standards into lint rules, LintMe facilitates better collaboration, onboarding, and reproducibility.
Future Directions: The paper suggests that this approach can be extended to complex documentation (manuals, tutorials), game instructions, and other culturally mediated texts. It also identifies the need for better knowledge base curation tools to lower the barrier for creating these rule sets.

Conclusion:
"Linting Style and Substance in READMEs" demonstrates that documentation linters can evolve beyond grammar checkers into powerful, customizable assistants for content quality. By blending programmatic precision with LLM-based semantic analysis, LintMe offers a scalable framework for maintaining high-quality, context-aware documentation across diverse software communities.