It feels like everywhere you turn these days, someone's talking about artificial intelligence. It's a word that's become a bit of a buzzword, especially in the public sector, and sometimes, I'll admit, it feels like it's being tossed around a little too casually.
Now, AI isn't exactly new. The concept has been around since the 1950s. But what's changed is how we typically encounter it. These days, when data scientists talk about AI, they're usually referring to systems built on machine learning models. These models are fascinating because they learn from data, figuring out their own rules to achieve a goal. This is quite different from what we might call traditional algorithms. Think of those older systems; they don't need to learn from data. They just follow a set of pre-programmed rules. We've seen these traditional algorithms used for ages in public services – remember that A-level exam results model that made headlines last summer? From an auditor's perspective, these are usually straightforward because the underlying logic is transparent. We're used to that.
But machine learning? That's a different beast. It's only been cautiously adopted in the public sector recently, and for good reason. Firstly, for these models to learn effectively, they need a lot of good quality data. And as we've seen in reports about data challenges across government, that's not always a given. Secondly, developing and deploying these models can be quite expensive, and the benefits aren't always immediate or guaranteed. In a world of tight public budgets, that's a tough sell.
Then there's the 'black box' element. It's not always clear from the outside what the machine will learn and, consequently, what decision-making rules it will generate. This makes it tricky to pinpoint the immediate benefits. A lot of the progress in machine learning has been in models where the decision-making rules are genuinely hard to understand or interrogate.
And finally, many of the decisions AI models would support involve people's personal lives – think health, benefits, or tax data. While data protection laws have gotten stronger, the organizational structures and clear accountabilities for using personal data in machine learning models aren't always in place. This puts public sector organizations at risk of accidentally falling short of evolving data protection standards.
Given all these hurdles, it's perhaps not surprising that in my public audit work, I haven't come across a huge number of machine learning models being used for decision-making. But they are out there, and it's likely we'll see more. That's why, working with audit colleagues in Norway, the Netherlands, Finland, and Germany, we've put together a white paper and an audit catalogue specifically on how to audit these machine learning models. You can find it at auditingalgorithms.net.
What we've identified are some key problem areas and risk factors. Developers often get laser-focused on optimizing specific numerical performance metrics, sometimes at the expense of crucial requirements like compliance, transparency, and fairness. It's also common for the people who build the models to be different from the ones who 'own' them in the decision-making process. If those 'product owners' don't clearly communicate their needs, you can end up with models that actually increase costs or make routine tasks more complicated, not less.
And then there's the reliance on external expertise. Many public sector organizations lack the in-house skills to build these applications themselves, so they turn to commercial providers. This can lead to them adopting a model without fully understanding how to maintain it or ensure it complies with all the relevant regulations.
For auditors, this means we need to be equipped. We need a solid grasp of the high-level principles behind machine learning. Understanding common coding languages and model implementations, and being able to use the right software tools, is becoming essential. And since machine learning often relies on powerful computing, which usually means cloud-based solutions, auditors need a basic understanding of cloud services too. It's a complex landscape, but one we're increasingly needing to navigate.
