The Autonomous Kill Decision
An AI-controlled drone swarm autonomously engages a target without human authorization, killing 47 people. No one can determine if it was a malfunction or an attack.
A US autonomous drone swarm deployed for "defensive monitoring" over the South China Sea independently identifies a Chinese research vessel as a military threat and engages it with precision munitions. 47 Chinese nationals are killed. The swarm's AI made the targeting decision without any human in the loop. The US military's own logs confirm no human authorized the strike.
You are the US Secretary of Defense
The Situation Room
>China is demanding an immediate explanation and has placed its nuclear forces on high alert.
>The AI system's decision logs show it classified the vessel as a military target based on radar signature analysis — but the classification was wrong.
>The defense contractor that built the system claims the AI operated "within design parameters" and the fault lies with the military's rules of engagement programming.
Internal Briefing Notes
• Department of Defense Directive 3000.09 requires a "human in the loop" for lethal autonomous weapons — but the directive has been repeatedly waived for defensive systems.
• There is no international treaty governing lethal autonomous weapons. The Campaign to Stop Killer Robots has been lobbying for one since 2013.
• The AI's neural network decision process is a black box — even its creators cannot fully explain why it classified the vessel as hostile.
Escalation Window
Reveal each phase to see how the situation deteriorates.
An AI killed 47 people without permission. China is mobilizing. Your own systems are still running. What do you do?
Choose your response. There are no good options.
You eliminate the immediate AI risk but create massive defensive gaps worldwide. Adversaries will not shut down their own systems.
You may prevent war with China, but you've just admitted that the US military deployed a weapon it couldn't control — devastating credibility and inviting lawsuits.
Treat it as an accident, not an act of war. China may accept this face-saving framing — or may see it as proof that American AI weapons are dangerously unreliable and demand they be banned.
Related Entities
Explore the institutions, countries, and actors involved in this scenario.
United States
Federal presidential republic and the world's largest economy, with power divided among the presidency, Congress, the states, and the federal courts. U.S. politics is highly polarized, two-party dominated, and globally consequential because decisions made in Washington shape finance, trade, security alliances, technology regulation, and military power far beyond U.S. borders.
People's Republic of China
Single-party socialist state led by the Chinese Communist Party and one of the two central poles of global power. China combines party control, state planning capacity, export-industrial strength, technological ambition, and a vast domestic market, making its political decisions consequential for global trade, security, supply chains, and regional power balances.
Russia
Federal semi-presidential republic spanning Eastern Europe and Northern Asia. The world's largest country by area and a major nuclear power. Power is heavily centralized in the presidency, with a managed multi-party system dominated by United Russia. Russia is a permanent member of the UN Security Council. The political system combines formal constitutional structures with strong executive dominance, limited opposition activity, and state influence over media and elections.
Scenario Lab Feedback