The Department for Work and Pensions (DWP) is facing criticism for its artificial intelligence system designed to detect welfare fraud. A recent internal review revealed significant disparities in how the system treats individuals based on age, disability, marital status, and nationality. This analysis, released under freedom of information laws to the Public Law Project, highlighted that the AI is more likely to flag certain groups for investigation.
Conducted in February 2024, the 11-page report indicated that the machine learning tool used to vet universal credit payments shows clear disparities across all protected characteristics evaluated. Despite these findings, the DWP reassured that these disparities don’t point to immediate discrimination or unfair treatment of specific individuals or groups. They emphasized that no automated decisions are made without human assessment, and all information is carefully considered by caseworkers.
Notably, the analysis did not assess whether other factors like race or sexual orientation contribute to unfair treatment, but DWP claims the safeguards in place apply to everyone. They intend to refine their analysis approach and plan to conduct further assessments every three months, evaluating the fairness and effectiveness of the current model.
Caroline Selman from the Public Law Project criticized the DWP’s approach, stating that there hasn’t been sufficient evaluation of whether the automated processes unfairly target marginalized communities. She called for a shift away from a reactive “hurt first, fix later” mindset.
The report remains vague on specifics, leaving unclear which age groups are most affected or how nationalities are treated. Likewise, there’s no clarity on whether disabled individuals are being disproportionately flagged compared to non-disabled individuals. The DWP defended its AI system, arguing that it does not replace human judgment; caseworkers still make final decisions.
While the assessment details measures to reduce potential bias, it overlooked the concept of “automation bias,” where users may overly trust computer-generated information. Attempts to gather insights on this issue from the DWP went unanswered before publication.
The scrutiny surrounding AI’s role in welfare systems isn’t isolated. Others have raised alarms about the impact of automated systems on marginalized groups. For instance, Amnesty International recently reported that Denmark’s automated welfare system hinders access for people with disabilities and low-income individuals. Similarly, investigations in Sweden revealed that automated processes disproportionately target vulnerable communities for fraud inquiries.