The irony of human control over AI agents

We rely on human control to keep AI agents safe. Yet that very oversight may erode the skills needed to exercise it well.

We rely on human control to keep AI agents safe. As long as somewhere a human in the loop watches, assesses and can intervene, so the thinking goes, we ultimately retain control. But that is exactly where an uncomfortable paradox arises: using the same AI systems that person is supposed to supervise can erode the skills required for good supervision.

The paradox of human in the loop

That is the central warning in the recent scientific position paper AI Agents Push Humans Out of the Loop by Margaret Mitchell, Avijit Ghosh and Samir Passi. The authors argue that the problem is not only that increasingly autonomous AI agents are harder to control. Their sharper point is that prolonged use of these systems can also impair the cognitive capacities of human supervisors. As an agent gathers more information, performs more analyses, makes more choices and handles more actions independently, the human has to think, weigh and decide less often. As a result, precisely the expertise we rely on when the system makes a mistake or encounters an exceptional situation may decline.

Not new: Ironies of Automation

Remarkably, this problem is not new at all. In 1983, psychologist and researcher Lisanne Bainbridge described the same mechanism in her now classic paper Ironies of Automation. Her observation was as simple as it was powerful: through automation we try to remove human error and limitations from a system, but ultimately we leave the human responsible for the situations that could not be automated. Those are usually the rare, complex and unexpected situations. At the same time, that same person has had fewer and fewer opportunities to practise the skills needed to act well at exactly such a moment.

That is the irony of automation: the better automation takes over normal work, the less prepared the human may become for the moment when human intervention is most needed.

A new dimension with Agentic AI

With Agentic AI, that old irony gains a new dimension. Traditional automation mainly took over actions and predictable process steps. AI agents can potentially also perform cognitive work: searching information, interpreting it, making plans, weighing alternatives, communicating with other systems and carrying out follow-up actions independently. This shifts the human further and further from executor to supervisor.

That seems efficient, but supervising work you barely do yourself is complicated. To recognise an error you must understand what a good outcome looks like. To correct a decision you must still have enough knowledge to formulate an alternative. And to intervene at a critical moment, you must have been mentally involved in what preceded it.

Human in the loop is no reassurance

Human in the loop thus becomes a far too simple reassurance. It is not about whether formally someone is still in the process. The relevant question is whether that person is still actually capable of meaningful oversight. Does that person have enough visibility into what the agent is doing? Does he or she understand why certain choices are made? Is critical thinking still regularly required? And do knowledge and skills remain sufficiently developed to push back against a system when necessary?

Mitchell, Ghosh and Passi therefore argue that supporting human cognitive capacities should be considered just as serious a part of AI agent design as improving the capabilities of the agent itself.

Consequences for productivity and organisations

This also has consequences for how organisations view productivity. The temptation is great to ask at every step: can AI take this over too? But perhaps we should also ask a second question: what happens to our organisation when people no longer do this themselves?

Not every human action that technically can be automated therefore has to disappear automatically. Sometimes it is wise to deliberately preserve moments of human involvement: making your own analysis before looking at the AI output, leaving certain decisions explicitly with people, letting employees practise without AI regularly, or designing systems so that not only the answer but also important intermediate steps remain visible. Not because AI could not do the work, but because the human must continue to be able to understand and judge it.

The fundamental design question

The fundamental design question for Agentic AI is therefore not how many people we can remove from the loop. It is what role we want people to retain in a world where more and more of that loop can be automated. Because if human control is ultimately our last line of defence, we cannot afford to slowly degrade that human into someone who only has to click a green checkmark. The greatest irony would be that we try to make AI safe with human control, while simultaneously building systems that hollow out the very capacity for human control.