5.4 Who's working on it, and where the case is weakest

The field has grown quickly but is still small, both compared with the scale of the problem and compared with the number of people working to make AI more capable. It needs a range of skills:

• Technical safety research: understanding what AI systems are doing internally, and making their behaviour reliable.
• Evaluations: testing systems for dangerous abilities before release, as METR does.
• Governance and policy: rules, standards, international agreements and the institutions to enforce them.
• Communication and field-building: explaining the problem accurately and bringing in people with the right skills.

Where the case is weakest.

• Disagreement. As the last lesson showed, estimates of catastrophic risk differ by an order of magnitude or more between groups of informed people. Some prominent researchers think worries about loss of control are overblown.
• Hype and incentives. AI companies benefit from people believing their products are extraordinarily powerful, so claims about capabilities deserve scrutiny.
• Tractability. It isn't clear how much safety research or regulation can change the outcome, especially if competition between companies and countries is intense. Some worry that safety work can end up speeding up the development of more capable systems.
• Timelines. Much of the urgency depends on how soon very capable systems arrive, and forecasts have often been wrong in both directions.