Using machine learning to build public policy agenda from social media conversations

Technical Summary: Building Public Policy Agendas from Social Media via Machine Learning

Problem Statement
The identification of issues and the subsequent setting of public policy agendas are critical, yet resource-intensive, phases of the policy-making process. Traditional methodologies for this stage rely heavily on manual data collection (e.g., opinion surveys, workshops) and qualitative analysis. These approaches are often laborious, costly to scale across large geographic areas, and prone to obsolescence due to the long cycles required for data collection, analysis, and validation. Furthermore, they require scarce human expertise, particularly in low-income contexts. While social media platforms like Twitter offer a vast, real-time repository of public opinion, there is a lack of research investigating the suitability of machine learning (ML) methods to systematically derive public policy agendas from these conversations, especially within Sub-Saharan Africa.

Methodology
The authors propose a five-stage, human-augmented machine learning pipeline designed to transform raw social media data into validated public policy agenda items. The methodology was implemented using a dataset of 56,737 tweets collected from Uganda during the 2020/2021 general elections.

1. Data Cleaning and Pre-processing: Raw tweets were cleaned to remove noise, including retweets, URLs, user handles, punctuation, non-English words, and stop words. The data was tokenized, stemmed, and vectorized to prepare for language modeling.
2. Keyword Extraction and Issue Identification: Two topic modeling (TM) techniques were employed to extract latent themes:
   • Latent Dirichlet Allocation (LDA): A traditional bag-of-words approach where the optimal number of topics was determined by maximizing coherence scores.
   • Top2Vec: A distributed representation method that automatically determines the number of topics without requiring stop-word lists or manual pre-processing.
   • From the resulting topics, 75 "issue-specific keywords" relevant to public policy (e.g., corruption, healthcare, security, education) were manually curated.
3. Narrative Creation (Text Generation): A Generative Pre-trained Transformer 2 (GPT-2) model (124M parameters) was fine-tuned on a dataset of 6,000 unique tweets associated with the curated keywords. The model was configured to generate short, coherent narratives (approx. 50 tokens) based on specific keyword prompts.
4. Narrative Validation: The quality of the generated text was evaluated by human raters (students with English majors) using a 4-point Likert scale measuring readability and coherence. Inter-rater agreement was calculated using Fleiss' Kappa.
5. Agenda Validation: The resulting agenda items were validated through two methods:
   • Similarity Analysis: Cosine similarity was computed between the GPT-2 generated texts and reference texts extracted from the manifestos of two major Ugandan political parties (NRM and NUP).
   • Human Evaluation: A crowd-sourced survey assessed whether the generated items represented valid social concerns for public policy attention.

Key Contributions

1. Proposed Framework: The paper introduces a semi-automated, human-augmented ML pipeline for building public policy agendas from social media data, demonstrating that ML can complement or validate agendas derived from traditional methods.
2. End-to-End Pipeline: The authors detail a complete workflow comprising five distinct stages: data cleaning, keyword extraction, narrative generation, narrative validation, and agenda validation.
3. Proof-of-Concept Implementation: A functional ML model was developed and tested using Twitter data from Uganda's 2020/2021 elections. The study provides empirical insight into social issues raised by the public during this period, specifically addressing the knowledge gap regarding ML applications for policy agenda setting in low-income countries.

Results

• Text Generation Quality: Human evaluation of the GPT-2 generated narratives yielded "very good" inter-rater agreement (IRA) for readability (Kappa = 0.78) and "good" agreement for coherence (Kappa = 0.60).
• Agenda Validation:
  • Similarity Analysis: The model achieved above-average cosine similarity scores (threshold > 0.5) for at least three out of five reference themes. Specifically, it showed strong alignment with reference texts regarding security (vs. NRM manifesto) and democracy/governance and education (vs. NUP manifesto). Performance was below average for economy and healthcare themes.
  • Human Evaluation: The crowd-sourced validation of agenda items resulted in "good" inter-rater agreement (Kappa = 0.45).
• Limitations Observed: The authors noted that while the model produced coherent text in many instances, some outputs were incoherent or lacked deep explanatory power regarding semantics and context. This was attributed to the inherent noise and lack of topic organization in raw social media datasets.

Significance and Claims
The paper claims that the proposed ML approach represents a "promising methodology" for identifying matters of public interest from social media conversations. It asserts that this approach offers a more efficient alternative for processing large quantities of policy-related input data compared to traditional manual surveys. However, the authors maintain a modest stance, acknowledging that the current implementation is a proof-of-concept. They explicitly state that the method did not fully meet expectations regarding text quality and that issues of data bias, fairness, and accountability must be considered before such approaches can be used as a basis for actual policy decision-making. The work does not claim to replace human expertise but rather to augment it, suggesting that further research is needed to improve text generation quality and explore automation in validation stages.