What human oversight requires once a system recommends

The recommendation is already highlighted in the queue. The person who is supposed to check it has a few minutes and a stack of cases. They accept it because the screen looks finished.

Article 14 of Regulation (EU) 2024/1689 is written for high-risk AI systems. It says those systems shall be designed and developed, including with appropriate human-machine interface tools, so natural persons can effectively oversee them while they are in use. Human oversight, in that article, aims to prevent or minimise the risks to health, safety or fundamental rights that may emerge when a system is used as intended, or under conditions of reasonably foreseeable misuse, in particular where those risks persist despite the other requirements in that section of the regulation.

What the assigned person has to be able to do

The article says the system shall be provided to the deployer so the natural persons to whom human oversight is assigned are enabled, as appropriate and proportionate, to do five things.

They have to properly understand the relevant capacities and limitations of the system, and be able to duly monitor its operation, including so they can detect and address anomalies, dysfunctions and unexpected performance. They have to remain aware of the possible tendency to automatically rely or over-rely on the output. The article names that tendency automation bias, and it singles out systems used to provide information or recommendations for decisions a natural person will take. They have to correctly interpret the output, taking into account the interpretation tools and methods available. They have to be able to decide, in a particular situation, not to use the system, or to disregard, override or reverse the output. They have to be able to intervene in the operation, or interrupt the system through a stop button or a similar procedure that lets the system come to a halt in a safe state.

I would treat that list as the job a named person can perform on a live case. If they can open the tool and still cannot say what it is bad at, the first ability is missing. If they cannot disregard the output, the highlight on the screen is doing the deciding.

Who builds the oversight, and who runs it

The article says the oversight measures shall be commensurate with the risks, the level of autonomy and the context of use. They are ensured in one or both of two ways. The provider identifies measures and, when technically feasible, builds them into the system before it is placed on the market or put into service. The provider also identifies measures that are appropriate for the deployer to implement. Assigning a person, and giving that person a way to monitor, interpret, override and stop, is the deployer's half.

For the systems listed at point 1(a) of Annex III, the article goes further. No action or decision is taken by the deployer on the basis of the identification unless at least two natural persons, with the necessary competence, training and authority, have separately verified and confirmed it.

What to write down for a live recommendation

The article is about high-risk systems. A company can still use the same five abilities as the test for any system that recommends a decision a person will own. The click is already happening.

1. Name the person assigned to oversight for one live system, and name the decision they are allowed to stop.
2. Write the capacities, the limitations and the anomalies they are watching for, in the same place as the task.
3. Give them a way to disregard, override or interrupt the output, and use it once on a case where the output is wrong.
4. Where the output is a recommendation a person will act on, ask whether the highlight has started to stand in for the decision. That is the automation bias the article names, already sitting in the queue.

An AI governance session is one place to assign that person and try the stop on a real case. The five abilities still have to show up on the next recommendation, whether or not the session is booked.