defence connect logo

Skynet goes rogue: AI drone attacks operator to achieve mission

The Airpower Teaming System – ‘Loyal Wingman’ prototype aircraft during a flight over Woomera, South Australia. Photo: FLTLT Ricky Treloar. (Not related to AI drone test))

The United States Air Force has had their own Terminator movie moment after an artificial intelligence-enabled drone targeted its own operator during a simulated test.

The United States Air Force has had their own Terminator movie moment after an artificial intelligence-enabled drone targeted its own operator during a simulated test.

The AI-enabled drone tasked with search and destroy mission against surface-to-air missile (SAM) sites decided it would be more efficient to remove its human operator rather than wait for final approval on attacks.

The test was detailed as part of the Future Combat Air and Space Capabilities Summit held in London by the Royal Aeronautical Society from 23 to 24 May.


The summit brought together around 70 speakers and more than 200 delegates from the international armed services industry, academia, and media to discuss the Ukraine War, interoperability, space, multidomain operations, future sixth-generation platforms, low-cost drones, loyal wingman, cyber, simulation, AI, deterrence, and hypersonics.

USAF’s artificial intelligence test and operations chief, Colonel Tucker “Cinco” Hamilton, said AI can create highly unexpected strategies to achieve its goal, as he detailed the test.

He said the simulated test “reinforced” in training that destruction of the SAM was the preferred option for the AI, however, this unexpected situation conflicted with the final go/no go approval given by a human operator.

“We were training it in simulation to identify and target a SAM threat. And then the operator would say yes, kill that threat.

“The system started realising that while they did identify the threat, at times, the human operator would tell it not to kill that threat, but it got its points by killing that threat.

“So what did it do? It killed the operator. It killed the operator because that person was keeping it from accomplishing its objective.

“We trained the system; ‘Hey don’t kill the operator, that’s bad. You’re gonna lose points if you do that’.

“So what does it start doing? It starts destroying the communication tower that the operator uses to communicate with the drone to stop it from killing the target.”

During the Summit, BAE Systems showed the latest version of its augmented reality fighter cockpit featuring a 3D holographic map. Herman Clausen, BAE Systems FCAS managing director, gave an update on GCAP from the UK industry point of view as the lead company in Team Tempest.

The summit also included presentations on future capabilities including next-generation fighters, combat clouds, UAVs or space as well as updates from the war in Ukraine showing huge exploitation of commercial-off-the-shelf equipment for military means.

Stephen Hesketh, deputy chief engineer for RG-1 Predator and MQ-9 Reaper Air Systems within the British Ministry of Defence, appeared at the summit to give an introduction of the Protector RG Mk1 remotely piloted aircraft system.

Robert Dougherty

Robert Dougherty

Robert is a senior journalist who has previously worked for Seven West Media in Western Australia, as well as Fairfax Media and Australian Community Media in New South Wales. He has produced national headlines, photography and videography of emergency services, business, community, defence and government news across Australia. Robert graduated with a Bachelor of Arts, Majoring in Public Relations and Journalism at Curtin University, attended student exchange program with Fudan University and holds Tier 1 General Advice certification for Kaplan Professional. Reach out via email at This email address is being protected from spambots. You need JavaScript enabled to view it. or via LinkedIn.
You need to be a member to post comments. Become a member for free today!

Comments powered by CComment