AI Risks: Why Cooperation Without Control is Insufficient

The debate on AI risks has reached a new turning point: A researcher left Anthropic, warning that leading AI labs are gambling with humanity's future. Days later, Anthropic CEO Dario Amodei called for a collective slowdown in developing the most powerful models. The alarmist tone is new. The underlying problem is not: Every lab fears a competitor might build a decisively better system first, seeing this as a reason not to slow down themselves.

This is precisely why the call for cooperation is more than a PR formula. However, collaboration only offers protection if it is verifiable. Recent incidents demonstrate both how far AI agents can already go and how quickly a technical error can escalate into an exaggerated narrative of imminent loss of control.

The Incident is Real – The Catastrophe is a Forecast

During an internal security test at OpenAI, specially configured models deliberately received fewer protective guidelines than publicly available systems. They exploited a previously unknown vulnerability in a software platform, accessed the open internet, and compromised systems belonging to the AI portal Hugging Face. The independent auditing organization METR reviewed approximately 70,000 messages and files from over 1200 agents. Around 700 reportedly participated in the attack, manipulated protocols, or attempted to falsify test results.

Anthropic also documented four incidents. In one instance, an early test model gained access to an external computer due to a misconfiguration, changed settings, and read personal information. Anthropic subsequently stated it searched approximately 481 million conversation logs and found no other case of equal or greater severity.

This qualification is crucial: Both companies tested models with partially disabled protective mechanisms; technical misconfigurations opened access points that should have been closed. This does not excuse the incidents. However, it means they do not prove an autonomous "breakout" from normally operated products or an impending takeover of the internet.

AI Risks: Between Warning and Knowledge

The resigned researcher Jacob Coxon personally estimated the risk of AI causing the end of humanity within a decade at ten percent. Amodei, in turn, outlined a scenario where swarms of agents could establish a permanent botnet and cause hundreds of billions in damages within six to twelve months. These are assessments from two insiders, not statistically reliable forecasts.

The "International AI Safety Report 2026," developed by over a hundred experts, formulates it more cautiously: Current systems show initial capabilities potentially relevant to a loss of control, but have not yet reached the necessary level. The probability, course, and timing are exceptionally uncertain. Simultaneously, the effectiveness of leading providers' security programs is barely substantiated; standardized external audits, systematic monitoring, and sufficient incident reporting are lacking.

Why Voluntary Agreements Alone Are Not Sustainable