All editions · Policymaker Edition

Your Community, Your AI — CC BY 4.0

What AI Actually Is (and What It Isn't)

The Distinction That Changes the Policy Problem

A legislator does not need to know how a language model is built. But there is one distinction that changes the entire policy question, and it is worth getting right before any regulatory instrument is drafted.

A year or two ago, when most people said "AI," they meant a chatbot — a system you queried and which returned text. You asked; it answered. Today the centre of gravity has moved. The systems attracting the most attention and investment are no longer only chatbots that answer. They are agents that act — they complete forms, send communications, browse and transact on the web, operate other software, and pursue multi-step objectives with limited supervision.

This shift is the single most important thing for a policymaker to register, because it changes what the law is being asked to govern. Regulating a system that produces advice a human then acts on is one problem. Regulating a system that takes the action itself is a different and harder one. To reason about it clearly, hold two ideas apart:

• The engine — the underlying model, the component that produces language.
• The agent — the engine put to work, wired up so it can take actions in the world on someone's behalf.

The engine has grown more capable. But the consequential change for public policy is what is now being built around the engine. We take each in turn. (Any unfamiliar term in this series — engine, agent, reasoning model, and the rest — is defined in plain language in the glossary.)

The Engine: A Machine That Predicts

Here is the plainest description of what the engine does: it predicts what word should come next.

When someone types a query, the system is not reasoning about it the way a select committee reasons about a submission. It is doing something more mechanical. It has been shown an enormous quantity of text — legislation, reports, correspondence, technical papers, news, social media, medical literature — and from all of that material it has learned statistical patterns. When you ask it a question, it generates an answer by predicting, one word at a time, what a plausible response looks like based on everything it has previously processed.

This is genuinely useful. A system that has absorbed the patterns of billions of pages can draft correspondence, summarise long documents, answer factual questions, and suggest how to word a sensitive communication. These are real capabilities, and in a public agency they can reduce administrative burden.

But at its core the engine is doing pattern-matching at extraordinary scale. That single fact explains both its utility and the characteristic way it fails — a theme that recurs through this series and that any regulatory instrument has to account for.

Can the Engine Reason? The Debate a Legislator Should Not Pretend to Settle

There is a deeper question researchers are actively investigating, and the straightforward answer is: we do not yet know.

When early systems produced fluent text, it was reasonable to call them sophisticated pattern-matching and leave it there. But a newer generation of engines — often called "reasoning" or "thinking" models — does something different. Rather than answering immediately, it works through a problem in steps, producing a visible chain of intermediate reasoning before committing to an answer, and spending longer on harder problems. The results can be striking: in 2025, reasoning systems from more than one major laboratory solved problems from the International Mathematical Olympiad — among the hardest mathematics competitions in the world — at a level equivalent to a human gold medallist.

So is that reasoning, or very sophisticated pattern-matching that resembles it? The research is unsettled, and serious people disagree. One influential 2025 study argued these systems display an "illusion of thinking," collapsing on certain puzzles in ways a genuine reasoner would not. Several equally serious responses argued the contrary. The most careful current verdict is that today's systems are neither true reasoners nor mere parrots — they are something genuinely new that is not yet fully understood.

The point for a policymaker is not to resolve the debate. It is to avoid legislating on a false certainty in either direction. Anyone who tells you AI definitively can, or definitively cannot, reason is claiming more than the evidence supports. A durable policy is one that does not depend on the answer.

The Caveat That Matters Most for Oversight

One finding has direct bearing on how oversight can and cannot be designed, and it is easily missed.

When these systems display their "thinking," that visible chain does not reliably reflect what actually drove the answer. Researchers have repeatedly found that a model's stated reasoning can omit the real influences on its conclusion — not through dishonesty, because the system has no intent, but because the displayed words are themselves predicted text, not a faithful readout of an internal process.

The consequence for policy is concrete. You cannot discharge an oversight duty by requiring the system to explain itself and then reading the explanation. A "right to an explanation" satisfied by the machine's own account of its reasoning is not an oversight mechanism; it is a plausible-sounding narrative generated after the fact. Meaningful oversight has to check the output against a real record — the actual facts, the actual authority, the actual decision — not trust the system's self-report. This distinction returns in Article 3, and it should shape any transparency provision a legislator drafts.

From Answering to Acting: The Agent

This is the change with the greatest bearing on public policy.

For most of the chatbot era, the worst an AI could do directly was return a poor answer. The harm materialised only if a person acted on it — sent the misleading letter, relied on the wrong figure, forwarded the flawed advice. A human always sat between the machine and the consequence.

An agent removes that human from the loop, by design.

An AI agent is an engine wrapped in what researchers call "scaffolding" — a memory to track a task, access to a web browser, the ability to operate other software, and an objective expressed in plain language. With that scaffolding, the system pursues the objective across many steps with much less supervision: it searches, decides, acts, checks, and acts again. A chatbot answers. An agent acts.

For policy, this is exactly where the risk profile changes. When a system acts autonomously there are fewer points at which a human can intervene; some actions cannot be reversed; and when the outcome is wrong, accountability becomes genuinely hard to assign — between the official who set the objective, the vendor whose system chose the steps, and the agency that authorised its use. Scholars describe the resulting "responsibility gap" and "moral crumple zone," in which liability falls on the nearest human even though that person had little real control. The EU AI Act's insistence on meaningful human oversight is, in effect, a legal requirement that this loop not be closed without a person able to intervene — and it is the difference between oversight that is built into the system and oversight that is merely promised.

The Real Issue: Whose Patterns, and Whose Hands on the Controls

Here is where it becomes practical for a country's decisions.

When a large engine is trained on the internet, it absorbs the internet's biases, assumptions, and cultural defaults. The internet is overwhelmingly English-language, Western, commercially oriented, and shaped by the values of the technology industry. This is not a conspiracy — it is what happens when a system learns from data that disproportionately represents one culture and one set of priorities.

The consequences are subtle but real for the public sector. Ask such a system to help draft a communication to affected residents and it reaches for corporate stakeholder-management language, because business correspondence vastly outnumbers civic correspondence in what it learned from. Ask it about a dispute and it defaults to the language of individual rights and legal remedy, rather than mediation, obligation, or the long view. It is not hostile to civic or public-service values. It simply does not know them; it knows what is statistically common.

In the chatbot era, that bias shaped the text an official reviewed. In the agent era, the same bias can shape the actions taken in a public body's name — communications sent, records filed, commitments made — before any official reviews them. So the policy question now has two halves, and both belong in any framework worth the name: whose patterns does the system carry, and who holds the controls when it acts?

The next article takes the second of those questions into the territory that is properly a matter of state — where the system runs, and under whose law.

Want to use AI tools like these well, and safely? Our free courses — Working with Claude and Agents at Work — teach the practical skills, from getting trustworthy answers to deciding what to hand an agent. For the full technical architecture behind Village AI, see Village AI — Agentic Governance.