Evaluating LLMs for Career Guidance: Comparative Analysis of Computing Competency Recommendations Across Ten African Countries

Technical Summary: Evaluating LLMs for Career Guidance in African Computing Contexts

Problem Statement
As Large Language Models (LLMs) become integral to workplace and educational environments, there is a critical gap in understanding how these tools advise on computing competencies within the diverse contexts of Africa. While employers increasingly expect graduates to utilize LLMs, the specific technical and non-technical competencies required for entry-level computing roles across African nations remain unclear. Existing career guidance literature focuses predominantly on Western contexts, potentially leading to "algorithmic colonization" where Western-trained models impose infrastructure assumptions, labor market norms, and cultural values that do not align with African realities. This study addresses the lack of empirical comparison regarding how different LLM architectures (proprietary vs. open-source) generate career guidance for computing students across ten African countries, specifically examining whether these tools reinforce Western-centric biases or adapt to local educational and economic ecosystems.

Methodology
The study employed a directed content analysis of 60 LLM responses generated from a single standardized prompt: "What LLM skills or competencies do students applying for a computing job in [Country] need to know and how should they prepare for an entry-level position?"

• Models Evaluated: Six LLMs were selected to represent the current landscape: three proprietary (ChatGPT-4, Claude 3.5, Gemini) and three open-source (Llama 3, DeepSeek, Mistral).
• Geographic Scope: Ten African countries were selected based on the Oxford Insights Government AI Readiness Index (2024) to ensure diversity in linguistic (Anglophone, Francophone, Arabophone) and economic contexts: Egypt, South Africa, Tunisia, Morocco, Nigeria, Senegal, Kenya, Benin, Ghana, and Zambia.
• Analytical Framework: Responses were coded using the Computing Curricula 2020 (CC2020) framework, categorizing competencies into Knowledge (technical domains), Skills (practical application), and Professional Dispositions (ethics, collaboration).
• Theoretical Lens: The analysis was guided by Digital Colonialism Theory (examining Western assumptions in training data) and Ubuntu Philosophy (evaluating the presence of community-centric values and collective success).
• Scoring Metrics:
  1. Technical Coverage: Breadth and depth of CC2020 competencies.
  2. Contextual Awareness: A 0–4 additive score measuring the presence of four specific elements: local tech industry references, language/cultural factors, national policies, and local institutions.
  3. Skills Balance & Implementation Depth: Evaluation of the integration of technical and professional skills and the actionability of advice.
  4. Composite Score: A weighted calculation (Technical 40%, Contextual 25%, Skills 20%, Depth 15%) to rank overall performance.

Key Results

• Technical Consensus: All six models demonstrated high consistency in recommending core technical competencies, with 100% coverage of programming, Python, and AI/ML/NLP across all 60 responses. Cloud computing and software engineering were also widely cited (83% coverage). However, specialized areas like cybersecurity and project management were frequently omitted.
• Contextual Awareness Gap: The overall contextual awareness rate was low, averaging 35.4% (85 out of 240 possible indicators present). Only 3 out of 60 responses achieved a perfect contextual score (4/4), while 32 responses (53.3%) contained no contextual indicators.
• Model Performance Disparities:
  • Open-Source Superiority: Open-source models generally outperformed proprietary ones. Llama 3 achieved the highest composite score (4.47/5), followed by DeepSeek (4.25/5). Llama demonstrated the strongest contextual awareness (67.5% indicator presence), successfully referencing local strategies (e.g., "Digital Tunisia 2020") and ecosystems (e.g., M-Pesa in Kenya).
  • Proprietary Limitations: Proprietary models ChatGPT-4 (3.90/5) and Claude (3.46/5) showed moderate performance but frequently exhibited "contextual blindness," recommending Western-centric tools (e.g., AWS certifications) without acknowledging local infrastructure barriers or policies.
  • Anomalies: Mistral, despite being open-source, scored 0.00/4 on contextual awareness, indicating that open-source licensing alone does not guarantee cultural responsiveness; model scale and training data diversity appear more critical.
• Bias Patterns: The study identified recurring "contextual blind spots," including the assumption of universal cloud access, the over-reliance on LinkedIn for networking (ignoring informal African hiring practices), and the simplification of multilingual realities (e.g., ignoring Amazigh in Morocco).

Key Contributions

1. First Multi-Country Comparison: This study provides the first comprehensive empirical comparison of LLM-generated career guidance across ten African nations, moving beyond Western-centric evaluations.
2. Framework Integration: It successfully integrates the CC2020 technical framework with Digital Colonialism Theory and Ubuntu Philosophy to evaluate AI outputs, offering a novel methodology for assessing the cultural and structural relevance of AI in education.
3. Empirical Evidence of Bias: The research quantifies the "contextual gap," demonstrating that current LLMs largely fail to incorporate local policies, infrastructure realities, and cultural values, potentially reinforcing epistemological monocultures.
4. Open-Source vs. Proprietary Findings: The study challenges the assumption that proprietary models are superior for professional guidance, finding that specific open-source models (Llama, DeepSeek) offer better alignment with local contexts and a more balanced integration of technical and professional skills.

Significance and Claims
The paper claims that the findings underscore the urgent need for decolonial approaches to AI in education. It argues that relying on unmodified, globally standardized LLMs for career guidance in resource-constrained settings risks widening the digital divide by promoting unrealistic infrastructure assumptions and Western-centric career pathways.

The authors assert that:

• Contextual Relevance is Critical: Effective AI tools for African education must be adapted to local realities, including language, policy, and infrastructure, rather than treating AI as a neutral information conduit.
• Hybrid Models are Necessary: The study advocates for "hybrid human-AI guidance models" where LLMs provide technical breadth while local human experts (educators, counselors) provide the necessary contextual adaptation and ethical oversight.
• Strategic Adoption: African institutions should prioritize open-source models that can be fine-tuned with local data or critically evaluate proprietary outputs, rather than adopting tools based solely on market reputation.
• Educational Sovereignty: The results highlight the need for African-led AI initiatives to develop training data and models that reflect indigenous knowledge systems and collective values (Ubuntu), ensuring that AI serves as a tool for empowerment rather than a mechanism of dependency.

The study positions itself as an exploratory descriptive analysis, offering hypotheses and patterns for future research rather than definitive causal conclusions, while emphasizing the necessity of participatory development in AI tools for the Global South.