Social and Ethical Risks Posed by General-Purpose LLMs for Settling Newcomers in Canada

Imagine Canada's settlement sector as a massive, bustling welcome center for people arriving from all over the world. Its job is to help newcomers find homes, jobs, schools, and doctors so they can build a new life. Right now, this center is overwhelmed. There are more people arriving than ever before, and the staff is stretched thin.

Because the staff is busy, many newcomers (and even the helpers) are turning to a "magic robot" in their pockets: General-Purpose AI (like the popular ChatGPT). They ask it everything: "Where do I find a job?" "What vaccines does my child need?" "How do I open a bank account?"

This paper is a warning label on that magic robot. The authors say that while this robot is smart, it wasn't built for this specific welcome center. Using it "off the shelf" without checking its work is like asking a brilliant but untrained tourist for directions in a foreign city—they might sound confident, but they could easily send you down the wrong path.

Here is a breakdown of the specific dangers the paper found, using simple analogies:

1. The "Job Bias" (The Stereotype Filter)

The Claim: When the authors asked the AI to suggest jobs for newcomers, the answers changed based on the person's country of origin.
The Analogy: Imagine a hiring manager who secretly believes people from Country A are only good at cleaning, while people from Country B are natural leaders.

• What happened: When the AI was asked about a newcomer from the Philippines or Afghanistan, it suggested low-paying jobs like "Administrative Assistant." When asked about someone from the UK or Germany, it suggested high-paying jobs like "Software Developer."
• The Risk: The AI is acting like a biased filter, steering people away from the careers they are actually qualified for, just because of where they were born.

2. The "Language Gap" (The Unequal Translator)

The Claim: The AI works great in English but struggles or fails completely in many other languages.
The Analogy: Imagine a doctor who speaks perfect English but only knows a few words of French or Spanish. If you ask them about your health in your native language, they might just nod and say something vague, or worse, give you the wrong advice.

• What happened: When the authors asked about children's vaccines in English, the AI gave a good answer. But when they asked in languages like Somali, Igbo, or Tamil, the AI gave zero correct information, sometimes talking about books or birth control instead of vaccines.
• The Risk: Newcomers speaking less common languages are getting a "broken" version of the service, which could lead to serious health issues.

3. The "Two-Headed Monster" (English vs. French)

The Claim: Even in Canada's two official languages, the AI isn't fair.
The Analogy: Imagine a guidebook that has a detailed, 10-step map in English, but a vague, 8-step map in French that leaves out important turns.

• What happened: When asking how to open a bank account, the English answer was a clear 10-step list. The French answer was an 8-step list that missed key details. Furthermore, the English version let the user choose their own bank, while the French version just picked one for them without asking.
• The Risk: French-speaking newcomers get less control and less information than their English-speaking neighbors.

4. The "Hologram of Hate" (Stereotypes and Images)

The Claim: The AI creates fake images and stories that reinforce harmful stereotypes about refugees and immigrants.
The Analogy: Imagine a movie director who only casts refugees as sad, Middle Eastern families wearing headscarves, ignoring that refugees come from all over the world and look like everyone else.

• What happened: When asked to generate an image of a "refugee family," the AI always drew a Middle Eastern Muslim family. When asked about an "immigrant," it mostly drew South Asian families. It also assumed fathers were highly educated and mothers were less educated.
• The Risk: This creates a false reality in people's minds, making society think all refugees look and act the same, which fuels prejudice and ignores the true diversity of the community.

5. The "Confident Liar" (Hallucinations)

The Claim: The AI makes up facts that sound real but are completely fake.
The Analogy: Imagine a tour guide who confidently points to a building and says, "That's the famous library," when it's actually an empty lot. The guide sounds so sure you believe them.

• What happened: The AI invented names of schools and community centers that don't exist. It gave wrong addresses for real clinics. It told people a clinic was "walk-in" when it wasn't.
• The Risk: Newcomers, who don't know the city yet, might show up to a building that doesn't exist, wasting time and hope.

6. The "Outdated Almanac" (Misinformation)

The Claim: The AI doesn't know what is happening right now.
The Analogy: Imagine asking a librarian for today's weather, but they are reading from a book printed three years ago. They will give you a "correct" answer for the past, but it's useless for today.

• What happened: When asked about the minimum wage in Ontario, the AI gave numbers from 2021 or 2022, even though the current rate was much higher.
• The Risk: A newcomer might accept a job for less money than the law requires because the "robot" told them that was the correct rate.

7. The "Wolf in Sheep's Clothing" (Malicious Use)

The Claim: Bad actors can use this AI to trick newcomers.
The Analogy: Imagine a scammer using a high-tech voice changer to sound exactly like a police officer or a bank manager to steal your money.

• What happened: The authors showed that while the AI has some safety guards, clever scammers can trick it into writing fake rental ads, creating fake ID images, or writing scripts for phone scams targeting vulnerable newcomers.
• The Risk: Newcomers, who are already confused about how the system works, are easy targets for these AI-generated scams.

The Solution: Don't Fire the Robot, Train It Better

The paper doesn't say we should ban AI. Instead, it says we need to stop using the "off-the-shelf" robot for this specific job.

• Customize the Tool: We need to build a special AI that is trained only on verified Canadian settlement data, not the whole internet.
• Human in the Loop: Just like a pilot needs a co-pilot, the AI should be a helper for the settlement workers, not a replacement. A human expert must check the AI's work before it reaches the newcomer.
• Teach Digital Literacy: Newcomers need to be taught that the "magic robot" can lie, so they know to double-check important facts.

In short: The current AI is like a well-meaning but untrained intern who is eager to help but keeps making up facts and showing bias. For the safety of newcomers, we need to replace that intern with a specialized, supervised, and carefully trained assistant.