Two forms of AI
The issues start with the technologies themselves. Broadly simplified, focus on two main AI types.
The first, “generative AI” like ChatGPT, generates probabilistic outputs predicting likely subsequent words based on training data. Although it mimics human-like responses, it merely reproduces existing patterns. Because of this, it can produce “hallucinations,” presenting false facts as truths.
These outputs aren’t strictly incorrect; the program functions as intended by generating contextually plausible word sequences. The fact that this issue cannot be resolved means the results can never be fully reliable.
The second type is “discriminative AI,” such as algorithms designed to identify patterns in data for decision-making purposes. These tools depend entirely on their input data and underlying algorithms, both crafted by humans and thus subject to human biases and errors.
Two recent Canadian examples highlight the negative impact of governments experimenting with generative AI and algorithmic decision-making in immigration and social assistance delivery.
In one instance, Immigration, Refugees and Citizenship Canada admitted using generative AI to deny a permanent residence application because the applicant’s job responsibilities supposedly did not match her Canadian work experience. The AI, however, inaccurately invented her current duties, causing the wrongful rejection.
In another case, Quebec initiated an AI-powered reform in 2025 of its social-assistance program called Project UNIR, employing algorithms to decide financial aid eligibility. This project removed “assigned agents” who guided clients throughout their cases, splitting workload among multiple officials in different regions instead.
One individual took his own life after staff falsely told him he wasn’t eligible for aid, a mistake worsened by the absence of a single human agent familiar with his file. Others have reported suicidal thoughts caused by administrative delays and lost paperwork within the system.
These two examples, alongside the three 2024 listeria deaths linked to the Canadian Food Inspection Agency’s use of flawed algorithms and data to select facilities for investigation, reveal troubling patterns previously witnessed in other contexts.
In 2016, Australia deployed an automated debt-recovery program known as Robodebt that sought to detect welfare fraud. This algorithmic system was so fraught with errors it mistakenly accused 450,000 people of fraud. Robodebt contributed to at least three suicides, police probes, a royal commission inquiry, and a compensation payout of AU$475 million.
The failed Australian experiment and these Canadian instances provide crucial warnings about relying on algorithms to manage public services while reducing frontline staffing.
Such failures share notable traits, as we discussed in our 2023 book The New Knowledge: Information, Data and the Remaking of Global Power.
One issue lies in technology quality and the fallout of automating public service functions.
Placing trust in generative AI to produce actionable reports is problematic because these systems struggle with real-world complexity and limit human workers’ ability to intervene and fix errors.
This opacity frustrates clients trying to understand decisions made about them, a problem worsened as overwhelmed phone lines prevent contact with human representatives. Quebec has invested millions to hire a private company to manage the overflow of calls.
A second concern relates to the workers interacting with or affected by these AI tools. To lessen risks from AI errors and hallucinations, governments have adopted a “human in the loop” approach, involving humans in overseeing algorithmic processes.
Still, this human-in-the-loop approach isn’t enough to avoid mistakes or harms. The mere presence of AI tools changes how personnel perform their jobs.
Automation often restricts frontline workers from applying their judgment and knowledge. Academics term this shift “ screen-level bureaucracy,” where bureaucracy evolves rather than disappears, becoming less accountable as algorithmic decisions are hidden behind proprietary technologies.
Moreover, the natural human tendency to see computer-generated results as authoritative is heightened when workers face increasing workloads and less time to scrutinize AI outputs carefully.
Finally, there is growing worry that dependence on AI will erode skills among civil servants, diminishing their capacity to identify and correct AI mistakes.
This further weakens the human-in-the-loop idea: the more AI is used, the more overall expertise declines.
Governments have also explored additional safeguards beyond human-in-the-loop.
For instance, Immigration, Refugees and Citizenship Canada states in its artificial intelligence strategy that AI is employed mainly for administrative tasks like summarizing and drafting documents, and that AI tools do not make final decisions on applications. IRCC labels AI in document handling as low risk, while AI that informs decision-makers is considered medium risk.
Nevertheless, this distinction between low and medium risk is meaningless if human rulings are influenced by inaccurate AI outputs leading to consequential mistakes.
Governments must accept that effectively integrating these technologies requires human oversight and evaluation at every stage.
Contrary to expectations of reducing labor expenses, a skilled and well-supported workforce remains essential to supervise AI tools properly. Without adequate time and authority, staff cannot thoroughly assess algorithmic outputs, increasing the chances of system failures and declining service quality.
The Carney administration bets heavily on AI to deliver cheaper government operations. While these technologies might offer benefits when applied selectively, starting from a preferred tech solution limits exploring alternative, possibly better approaches. Neither current AI technology nor the many documented problematic outcomes predict anything but gradual harm to Canadians and public institutions.
Original article: thetyee.ca
