5.3 What could go wrong

The International AI Safety Report 2026 groups the risks from advanced AI into three types.

Misuse. People using AI to cause harm on purpose: cyberattacks, scams and deepfakes, or help with developing biological weapons (see Unit 4). As systems get more capable, the barrier to entry for such attacks falls, and the damage a single attacker can do grows.

Malfunction, including loss of control. Systems that don't do what their developers intended. Most such failures are mundane. The more worrying ones involve AI agents that pursue goals with little human oversight. The report notes that in tests, models have "disabled simulated oversight mechanisms" and then made false statements to justify what they did. Today's systems are not about to take over. The concern is about what happens as they become more capable than the people supervising them, and harder to check.

Systemic risks. Effects that come from AI being everywhere rather than from any single failure. These include disruption to labour markets, erosion of people's autonomy, and power concentrating in the hands of whoever controls the most capable systems.

80,000 Hours, the careers organisation, adds a point about scale. Once you have one AI system that can do a skilled job, you can run many copies of it at once. This is part of why some researchers expect progress, and the risks with it, to accelerate.

How likely are the worst outcomes? Here, more than anywhere in this course, informed people disagree.

The gap between these groups is itself informative. Superforecasters have strong track records on shorter-term questions; AI experts know the technology best. Neither group has a track record on questions like this one.