Concordance Comparison as a Means of Assembling Local Grammars

Technical Summary: Concordance Comparison as a Means of Assembling Local Grammars

Problem Statement
Named Entity Recognition (NER) for person names is a critical yet challenging task in information extraction, primarily because person names constitute an open word class that expands continuously. While machine learning and hybrid approaches are common, rule-based methods using Local Grammars (LGs) remain effective, particularly for Portuguese. However, constructing LGs requires significant linguistic expertise and manual effort, with limited computational tools available to assist in the selection and combination of multiple small grammars. The specific challenge addressed is how to systematically assemble a set of small, individual Local Grammar Graphs (LGGs) into a cohesive, high-performing grammar without relying solely on trial-and-error or purely manual intuition.

Methodology
The authors propose a methodology that utilizes the Unitex open-source system and its ConcorDiff concordance comparison tool to aid in the manual composition of Local Grammars. The process involves the following steps:

1. Input Repository: A collection of small LGGs designed to recognize person names in Portuguese texts (some derived from literature, others created by the authors) serves as the starting point.
2. Concordance Generation: Each LGG is applied to the "Golden Collection" (GC) of the Second HAREM corpus (129 annotated texts containing 1,609 person name entities). This generates a concordance file for each grammar, listing recognized occurrences with context.
3. Pairwise Comparison: The ConcorDiff tool compares concordance files pairwise. It visualizes differences using an HTML interface where:
   • Blue lines indicate identical occurrences found in both grammars.
   • Red lines indicate overlapping but non-identical occurrences (e.g., one grammar recognizes a full name, the other only a part).
   • Green/Pink lines indicate occurrences unique to one grammar.
4. Set-Theoretic Analysis: The authors analyze these comparisons to identify set-theoretic relationships (inclusion, intersection, and disjunction) between the sets of occurrences identified by each grammar pair.
5. Grammar Assembly: Based on these relationships (summarized in Table 1 of the paper), the authors apply specific logic to decide which grammars to retain, discard, or combine. For instance, if Grammar A's results are a subset of Grammar B's, Grammar A may be discarded; if they are disjoint, both are retained. This process resulted in a final, larger grammar composed of 30 LGGs.
6. Refinement: Manual adjustments were made to handle specific Portuguese linguistic features, such as including honorific titles (e.g., "Rainha") and social positions in lowercase within the named entity tags, based on HAREM annotation guidelines.

Key Contributions

• Concordance Comparison as a Design Tool: The paper introduces a systematic approach using concordance comparison to visualize the relationships between different grammars, transforming the assembly process from a purely manual task into one guided by observable set-theoretic data.
• Decision Framework: The authors define a set of rules (Table 1) linking specific concordance comparison patterns (e.g., inclusion vs. disjunction) to decisions on which grammars to keep or discard.
• Improved Portuguese NER: The methodology was used to construct an enhanced LG for Portuguese person name extraction that outperformed the state-of-the-art system (Rembrandt) for the Second HAREM evaluation.

Results
The final assembled Local Grammar was evaluated against the Second HAREM Golden Collection using the SAHARA evaluation system.

• Comparison with Rembrandt: When evaluating the 'person' category, specifically the 'individual' subtype (PERSON(INDIVIDUAL)), the proposed LG achieved an F-Measure of 76.86%, compared to Rembrandt's 70.76%. This represents a 6-point gain over the previous best system.
• Precision and Recall: The final LG achieved 79.75% precision and 74.18% recall. The recall was notably higher (approx. 10 points) than Rembrandt, though precision was slightly higher as well.
• Broader Scope: When evaluated against other tools for all 'person' subtypes (PERSON(*)), the LG achieved an F-Measure of 69% (Precision 81%, Recall 60%). The authors note that while precision is superior, recall is lower than some hybrid systems because the LG currently recognizes only two subtypes ('individual' and 'position') rather than the eight subtypes recognized by broader systems.

Significance and Claims
The paper claims that concordance comparison offers a practical, computational aid for the manual construction of Local Grammars. By providing an overview of what each grammar recognizes within a specific corpus, the tool allows researchers to identify ambiguities, false positives, and redundancy efficiently. The authors assert that this approach produced a "more compact and easily understandable grammar" and demonstrated that LGs, when properly assembled using this method, can surpass state-of-the-art rule-based and hybrid systems for specific NER tasks in Portuguese. The work suggests that this methodology is transferable to other NER tasks and languages, and future work may involve automating the creation of elementary LGGs from examples, using concordance comparison to facilitate decision-making in that automation.