AI decision-making in welfare, the questions is not when, but how

New Zealand just legislated AI decision-making in welfare. How can we make sure this does more good than harm? 

I have spent years now working on digital change in government. And my biggest takeaway from that work is not about technology. It is how you implement something matters as much as what you implement.

We know that AI makes mistakes. Not because it's bad, but because all technology has limits. It works from patterns in historical data. It doesn't understand context the way a skilled experienced professional does. It can't weigh up personal circumstances against a rule that wasn't written with their specific situation in mind.

When those limits go unchecked at scale, the consequences are real. We've seen it in Australia where Robodebt debt recovery scheme that falsely accused hundreds of thousands of people, in US where nearly 48,000 false fraud accusations, issued automatically with no human review, and in Europe, where automated welfare systems have encoded racial and ethnic bias directly into benefit decisions.

In every case, the technology wasn't the only failure. The transformation process also was.

For us, change managers, AI implementation in government demands something more than a standard rollout. It requires us to ask harder questions:

• Do frontline staff understand what the system is doing and why? Not just how to use it but enough to know when to question it and how.
• Have we designed the system for failure, not just success? What happens when the system is wrong? Is there a tested, human-led process to catch it, correct it, and reach the person affected?
• Has the system been tested against the full diversity of the population it serves or only optimised for the majority case?
• Is 'human oversight' defined in writing? Who reviews what, how often, and what triggers escalation? Vague assurances are not safeguards.
• Who is accountable when the algorithm gets it wrong? That answer needs to be a name, not a process.
• Are we measuring the right outcomes? Automation makes it easy to count decisions per day. But are we tracking accuracy? Are we hearing from frontline staff when something feels off? How do we track and, more importantly, respond to that?

Government digital transformation can genuinely improve public services. Reducing administrative burden on frontline workers is a real and worthy goal. But rushed implementation, skipped consultation, and vague accountability structures are how well-intentioned technology causes harm to the people it was meant to help.

Our role as change managers here isn't just to make the transition smooth. It's to make it safe, to be the people in the room asking the uncomfortable questions before go-live, not after. That's the work worth doing.